Report on engine updates

[mbercx]

@sphuber I decided to write down my experiences with your changes in the engine update branch here, since there are more formatting options than on Slack, and it can also be used by @unkcpz to comment in one (persistent) location.

I'm not sure if this is related to the recent changes, but I’ve had some calculations running before shutting down my mac Mini to move it. Now I’m trying to restart them properly, but am running into a few issues.

Note: the calculations have all finished on the remote machine.

Pausing processes

First thing: if I try to pause the processes while the daemon is not running, a nice warning is shown, but an error is also not caught:

❯ verdi process pause -a
Warning: daemon is not running, so process may not be reachable
Traceback (most recent call last):
  File "/Users/mbercx/.virtualenvs/super/bin/verdi", line 8, in <module>
    sys.exit(verdi())
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mbercx/project/super/code/aiida-core/aiida/cmdline/utils/decorators.py", line 73, in wrapper
    return wrapped(*args, **kwargs)
  File "/Users/mbercx/project/super/code/aiida-core/aiida/cmdline/utils/decorators.py", line 148, in wrapper
    return wrapped(*args, **kwargs)
  File "/Users/mbercx/project/super/code/aiida-core/aiida/cmdline/commands/cmd_process.py", line 247, in process_pause
    control.pause_processes(processes, all_entries=all_entries, timeout=timeout, wait=wait, message=message)
  File "/Users/mbercx/project/super/code/aiida-core/aiida/engine/processes/control.py", line 131, in pause_processes
    raise DaemonException('The daemon is not running.')
aiida.engine.daemon.client.DaemonException: The daemon is not running.

Stopping and restarting the daemon seems to be going very smoothly so far.
However, if I try and pause the processes that are running, it claims all the processes have been paused:

❯ verdi process pause -a
Report: paused Process<17843>
Report: paused Process<12333>
Report: paused Process<12982>
Report: paused Process<12231>
Report: paused Process<12985>
Report: paused Process<17009>
Report: paused Process<12859>
Report: paused Process<16934>
Report: paused Process<17990>
Report: paused Process<16917>
Report: paused Process<17847>

But this is definitely not the case:

❯ verdi process list
   PK  Created    Process label      Process State    Process status
-----  ---------  -----------------  ---------------  ---------------------------------------
12231  6D ago     EpwWorkChain       ⏵ Waiting
12333  6D ago     EpwWorkChain       ⏵ Waiting
12859  6D ago     PhBaseWorkChain    ⏵ Waiting
12982  6D ago     PhBaseWorkChain    ⏵ Waiting
12985  6D ago     PhCalculation      ⏵ Waiting        Monitoring scheduler: job state RUNNING
16917  4D ago     SuperConWorkChain  ⏵ Waiting
16934  4D ago     SuperConWorkChain  ⏵ Waiting
17009  4D ago     SuperConWorkChain  ⏵ Waiting
17843  3D ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state RUNNING
17847  3D ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state RUNNING
17990  3D ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state RUNNING

Total results: 11

Report: last time an entry changed state: 3D ago (at 10:31:41 on 2023-06-01)
Report: Checking daemon load... OK
Report: Using 3% of the available daemon worker slots.

If I stop the daemon and analyse the RabbitMQ tasks:

❯ verdi daemon stop
Profile: dev
Stopping the daemon... OK
❯ verdi devel rabbitmq tasks analyze
Success: No inconsistencies detected between database and RabbitMQ.

All seems well, but since it's clearly still blocked tried to revive it:

❯ verdi devel rabbitmq tasks revive 12985
Warning: This command should only be used if you are absolutely sure the process task was lost.
Do you want to continue? [y/N]: y
Warning: Failed to deliver message to Process<12985>: <Future at 0x133d1f9a0 state=pending>|None

But that unfortunately also failed.

Note that these issues may be entirely unrelated to the changes in your branch. I think simply shutting down the system as the daemon is running may have caused some of these troubles. Simply reporting them here along with my other experiences.

Restarting the daemon

I've also tested the issue I was experiencing in #6041, and can still reproduce it. I basically:

1. Start a work chain which submits a process that does a (rather inefficient) copy of some big files during the upload transport step.
2. Try to restart the daemon during the upload step.

[2] fails:

❯ verdi daemon restart
Restarting the daemon... FAILED
Critical: Connection to the daemon timed out.

With approx. 30 of the following errors, spaced out by one second, in the circus logs:

Traceback (most recent call last):
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/tornado/ioloop.py", line 919, in _run
    val = self.callback()
  File "/Users/mbercx/.virtualenvs/super/lib/python3.9/site-packages/circus/util.py", line 1038, in wrapper
    raise ConflictError("arbiter is already running %s command"
circus.exc.ConflictError: arbiter is already running watcher_restart command

[sphuber]

@sphuber I decided to write down my experiences with your changes in the engine update branch here, since there are more formatting options than on Slack, and it can also be used by @unkcpz to comment in one (persistent) location.

Thanks a lot @mbercx . Very useful.

First thing: if I try to pause the processes while the daemon is not running, a nice warning is shown, but an error is also not caught:

This is unrelated and also happens on main. I will push a fix.

Stopping and restarting the daemon seems to be going very smoothly so far. However, if I try and pause the processes that are running, it claims all the processes have been paused:
But this is definitely not the case:

This also should be unrelated. I think the text in the reponse of the verdi process pause is not very accurate. What it probably should say is something a request to pause the process has been sent. But this still needs to be processed by the daemon worker, and the process may not actual go into the paused state until some later time, depending on how busy the worker is.

If I stop the daemon and analyse the RabbitMQ tasks:
All seems well, but since it's clearly still blocked tried to revive it:

Naughty naughty! The analysis command strictly said there were no inconsistencies, so launching revive is guaranteed to cause problems. That command should actually no longer be used manually tbh. If there is a problem, the analyze command will detect and report it and suggest it is rerun with the --fix flag, and that will automatically do the right thing.

❯ verdi devel rabbitmq tasks revive 12985
Warning: This command should only be used if you are absolutely sure the process task was lost.
Do you want to continue? [y/N]: y
Warning: Failed to deliver message to Process<12985>: <Future at 0x133d1f9a0 state=pending>|None

This is also probably the case on main. None of this functionality should have been affected by the changes.

Note that these issues may be entirely unrelated to the changes in your branch. I think simply shutting down the system as the daemon is running may have caused some of these troubles. Simply reporting them here along with my other experiences.

So yes, most if not all of these cases are "problems" that you will also come across on main. Essentially, (with the exception of the first) they are mostly due to what happens when the daemon is very busy. In that case it cannot respond instantaneously. Not sure what else we can do to improve this, other than some of the language used in the responses suggesting more that the daemon will get to it once it can.

So then on to what I hope the branch should actually be changing: the updates in the lower lying libraries should hopefully mostly improve connection stability for RabbitMQ. Currently, processes can except because the connection to RabbitMQ is lost and not correctly re-established. Classical examples are these ChannelInvalidStateError: writer is None exceptions.

The main thing I am looking for is that you can run your typical workflows, without any "new" errors, because we know there are some problems on main related to running high-throughput (most notably #4876 ). So please run test some more with "heavy'ish" loads and report if any of the processes except.

[sphuber]

Aha, I think I see the problem. As you can see, from the checkpoint, process 12985 is actually paused (it has the _paused attribute). However, the node actually has a state that seems not paused. Now when you do verdi process play -a, the -a flag ensures a query is performed to find all paused nodes, but that is done on the paused attribute on the node. So node 12985 gets skipped in this query.

I bet that if you run verdi process play 12985 it will start moving again. I have seen this problem before and essentially it happens when a process is paused and so the Process instance is paused, but the proxy on the ProcessNode is not properly updated. Given that most of the code checks the ProcessNode to determine its state, if it is inconsisten with the Process, there will be problems like this.

It is very likely that this indiscrepancy is possible if you shut the daemon down hard. Imagine it was just busy pausing a process, and right after it paused it but before it got to chance to update the node, it was killed.

Maybe as a fix, we can have a "consistency" check when a process is reconstructed from the checkpoint to check whether node attributes are insync.

[mbercx]

I bet that if you run verdi process play 12985 it will start moving again.

I was rooting that I would be able to post an "It's alive!" gif, but alas

❯ verdi process play 12985
Error: Process<12985> is unreachable.

Reran the command with verbosity level "info", logs below!

Full "Info" logs

 verdi process play -v info 12985
-Info: (Obj|db_object) _configure_property(id, Column)
Info: (Obj|db_object) _configure_property(hashkey, Column)
Info: (Obj|db_object) _configure_property(compressed, Column)
Info: (Obj|db_object) _configure_property(size, Column)
Info: (Obj|db_object) _configure_property(offset, Column)
Info: (Obj|db_object) _configure_property(length, Column)
Info: (Obj|db_object) _configure_property(pack_id, Column)
Info: (Obj|db_object) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_object>, primary_key=True, nullable=False)])
Info: (Obj|db_object) constructed
\Info: (DbSetting|db_dbsetting) _configure_property(id, Column)
Info: (DbSetting|db_dbsetting) _configure_property(key, Column)
Info: (DbSetting|db_dbsetting) _configure_property(val, Column)
Info: (DbSetting|db_dbsetting) _configure_property(description, Column)
Info: (DbSetting|db_dbsetting) _configure_property(time, Column)
Info: (DbSetting|db_dbsetting) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbsetting>, primary_key=True, nullable=False)])
Info: (DbSetting|db_dbsetting) constructed
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(_metadata, Column)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(aiidauser, RelationshipProperty)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(dbcomputer, RelationshipProperty)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(id, Column)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(aiidauser_id, Column)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(dbcomputer_id, Column)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(auth_params, Column)
Info: (DbAuthInfo|db_dbauthinfo) _configure_property(enabled, Column)
Info: (DbAuthInfo|db_dbauthinfo) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbauthinfo>, primary_key=True, nullable=False)])
Info: (DbAuthInfo|db_dbauthinfo) constructed
Info: (DbComputer|db_dbcomputer) _configure_property(_metadata, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(id, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(uuid, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(label, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(hostname, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(description, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(scheduler_type, Column)
Info: (DbComputer|db_dbcomputer) _configure_property(transport_type, Column)
Info: (DbComputer|db_dbcomputer) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbcomputer>, primary_key=True, nullable=False)])
Info: (DbComputer|db_dbcomputer) constructed
Info: (DbUser|db_dbuser) _configure_property(id, Column)
Info: (DbUser|db_dbuser) _configure_property(email, Column)
Info: (DbUser|db_dbuser) _configure_property(first_name, Column)
Info: (DbUser|db_dbuser) _configure_property(last_name, Column)
Info: (DbUser|db_dbuser) _configure_property(institution, Column)
Info: (DbUser|db_dbuser) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbuser>, primary_key=True, nullable=False)])
Info: (DbUser|db_dbuser) constructed
Info: (DbComment|db_dbcomment) _configure_property(dbnode, RelationshipProperty)
Info: (DbComment|db_dbcomment) _configure_property(user, RelationshipProperty)
Info: (DbComment|db_dbcomment) _configure_property(id, Column)
Info: (DbComment|db_dbcomment) _configure_property(uuid, Column)
Info: (DbComment|db_dbcomment) _configure_property(dbnode_id, Column)
Info: (DbComment|db_dbcomment) _configure_property(ctime, Column)
Info: (DbComment|db_dbcomment) _configure_property(mtime, Column)
Info: (DbComment|db_dbcomment) _configure_property(user_id, Column)
Info: (DbComment|db_dbcomment) _configure_property(content, Column)
Info: (DbComment|db_dbcomment) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbcomment>, primary_key=True, nullable=False)])
Info: (DbComment|db_dbcomment) constructed
Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(id, Column)
Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(dbnode_id, Column)
Info: (DbGroupNode|db_dbgroup_dbnodes) _configure_property(dbgroup_id, Column)
Info: (DbGroupNode|db_dbgroup_dbnodes) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbgroup_dbnodes>, primary_key=True, nullable=False)])
Info: (DbGroupNode|db_dbgroup_dbnodes) constructed
Info: (DbGroup|db_dbgroup) _configure_property(user, RelationshipProperty)
Info: (DbGroup|db_dbgroup) _configure_property(dbnodes, RelationshipProperty)
Info: (DbGroup|db_dbgroup) _configure_property(id, Column)
Info: (DbGroup|db_dbgroup) _configure_property(uuid, Column)
Info: (DbGroup|db_dbgroup) _configure_property(label, Column)
Info: (DbGroup|db_dbgroup) _configure_property(type_string, Column)
Info: (DbGroup|db_dbgroup) _configure_property(time, Column)
Info: (DbGroup|db_dbgroup) _configure_property(description, Column)
Info: (DbGroup|db_dbgroup) _configure_property(extras, Column)
Info: (DbGroup|db_dbgroup) _configure_property(user_id, Column)
Info: (DbGroup|db_dbgroup) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbgroup>, primary_key=True, nullable=False)])
Info: (DbGroup|db_dbgroup) constructed
Info: (DbLog|db_dblog) _configure_property(_metadata, Column)
Info: (DbLog|db_dblog) _configure_property(dbnode, RelationshipProperty)
Info: (DbLog|db_dblog) _configure_property(id, Column)
Info: (DbLog|db_dblog) _configure_property(uuid, Column)
Info: (DbLog|db_dblog) _configure_property(time, Column)
Info: (DbLog|db_dblog) _configure_property(loggername, Column)
Info: (DbLog|db_dblog) _configure_property(levelname, Column)
Info: (DbLog|db_dblog) _configure_property(dbnode_id, Column)
Info: (DbLog|db_dblog) _configure_property(message, Column)
Info: (DbLog|db_dblog) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dblog>, primary_key=True, nullable=False)])
Info: (DbLog|db_dblog) constructed
Info: (DbNode|db_dbnode) _configure_property(dbcomputer, RelationshipProperty)
Info: (DbNode|db_dbnode) _configure_property(user, RelationshipProperty)
Info: (DbNode|db_dbnode) _configure_property(outputs_q, RelationshipProperty)
Info: (DbNode|db_dbnode) _configure_property(id, Column)
Info: (DbNode|db_dbnode) _configure_property(uuid, Column)
Info: (DbNode|db_dbnode) _configure_property(node_type, Column)
Info: (DbNode|db_dbnode) _configure_property(process_type, Column)
Info: (DbNode|db_dbnode) _configure_property(label, Column)
Info: (DbNode|db_dbnode) _configure_property(description, Column)
Info: (DbNode|db_dbnode) _configure_property(ctime, Column)
Info: (DbNode|db_dbnode) _configure_property(mtime, Column)
Info: (DbNode|db_dbnode) _configure_property(attributes, Column)
Info: (DbNode|db_dbnode) _configure_property(extras, Column)
Info: (DbNode|db_dbnode) _configure_property(repository_metadata, Column)
Info: (DbNode|db_dbnode) _configure_property(dbcomputer_id, Column)
Info: (DbNode|db_dbnode) _configure_property(user_id, Column)
Info: (DbNode|db_dbnode) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dbnode>, primary_key=True, nullable=False)])
Info: (DbNode|db_dbnode) constructed
Info: (DbLink|db_dblink) _configure_property(input, RelationshipProperty)
Info: (DbLink|db_dblink) _configure_property(output, RelationshipProperty)
Info: (DbLink|db_dblink) _configure_property(id, Column)
Info: (DbLink|db_dblink) _configure_property(input_id, Column)
Info: (DbLink|db_dblink) _configure_property(output_id, Column)
Info: (DbLink|db_dblink) _configure_property(label, Column)
Info: (DbLink|db_dblink) _configure_property(type, Column)
Info: (DbLink|db_dblink) Identified primary key columns: ColumnSet([Column('id', Integer(), table=<db_dblink>, primary_key=True, nullable=False)])
Info: (DbLink|db_dblink) constructed
Info: select pg_catalog.version()
Info: [raw sql] {}
Info: select current_schema()
Info: [raw sql] {}
Info: show standard_conforming_strings
Info: [raw sql] {}
Info: BEGIN (implicit)
Info: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
Info: [generated in 0.00008s] {'name': 'alembic_version'}
Info: Context impl PostgresqlImpl.
Info: Will assume transactional DDL.
Info: select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
Info: [cached since 0.0125s ago] {'name': 'alembic_version'}
Info: SELECT alembic_version.version_num 
FROM alembic_version
Info: [generated in 0.00005s] {}
Info: (DbSetting|db_dbsetting) _post_configure_properties() started
Info: (DbSetting|db_dbsetting) initialize prop id
Info: (DbSetting|db_dbsetting) initialize prop key
Info: (DbSetting|db_dbsetting) initialize prop val
Info: (DbSetting|db_dbsetting) initialize prop description
Info: (DbSetting|db_dbsetting) initialize prop time
Info: (DbSetting|db_dbsetting) _post_configure_properties() complete
Info: (DbAuthInfo|db_dbauthinfo) _post_configure_properties() started
Info: (DbAuthInfo|db_dbauthinfo) initialize prop _metadata
Info: (DbAuthInfo|db_dbauthinfo) initialize prop aiidauser
Info: DbAuthInfo.aiidauser setup primary join db_dbuser.id = db_dbauthinfo.aiidauser_id
Info: DbAuthInfo.aiidauser setup secondary join None
Info: DbAuthInfo.aiidauser synchronize pairs [(db_dbuser.id => db_dbauthinfo.aiidauser_id)]
Info: DbAuthInfo.aiidauser secondary synchronize pairs []
Info: DbAuthInfo.aiidauser local/remote pairs [(db_dbauthinfo.aiidauser_id / db_dbuser.id)]
Info: DbAuthInfo.aiidauser remote columns [db_dbuser.id]
Info: DbAuthInfo.aiidauser local columns [db_dbauthinfo.aiidauser_id]
Info: DbAuthInfo.aiidauser relationship direction symbol('MANYTOONE')
Info: (DbUser|db_dbuser) _configure_property(authinfos, RelationshipProperty)
Info: DbUser.authinfos setup primary join db_dbuser.id = db_dbauthinfo.aiidauser_id
Info: DbUser.authinfos setup secondary join None
Info: DbUser.authinfos synchronize pairs [(db_dbuser.id => db_dbauthinfo.aiidauser_id)]
Info: DbUser.authinfos secondary synchronize pairs []
Info: DbUser.authinfos local/remote pairs [(db_dbuser.id / db_dbauthinfo.aiidauser_id)]
Info: DbUser.authinfos remote columns [db_dbauthinfo.aiidauser_id]
Info: DbUser.authinfos local columns [db_dbuser.id]
Info: DbUser.authinfos relationship direction symbol('ONETOMANY')
Info: DbUser.authinfos lazy loading clause :param_1 = db_dbauthinfo.aiidauser_id
Info: DbAuthInfo.aiidauser lazy loading clause db_dbuser.id = :param_1
Info: DbAuthInfo.aiidauser will use Session.get() to optimize instance loads
Info: (DbAuthInfo|db_dbauthinfo) initialize prop dbcomputer
Info: DbAuthInfo.dbcomputer setup primary join db_dbcomputer.id = db_dbauthinfo.dbcomputer_id
Info: DbAuthInfo.dbcomputer setup secondary join None
Info: DbAuthInfo.dbcomputer synchronize pairs [(db_dbcomputer.id => db_dbauthinfo.dbcomputer_id)]
Info: DbAuthInfo.dbcomputer secondary synchronize pairs []
Info: DbAuthInfo.dbcomputer local/remote pairs [(db_dbauthinfo.dbcomputer_id / db_dbcomputer.id)]
Info: DbAuthInfo.dbcomputer remote columns [db_dbcomputer.id]
Info: DbAuthInfo.dbcomputer local columns [db_dbauthinfo.dbcomputer_id]
Info: DbAuthInfo.dbcomputer relationship direction symbol('MANYTOONE')
Info: (DbComputer|db_dbcomputer) _configure_property(authinfos, RelationshipProperty)
Info: DbComputer.authinfos setup primary join db_dbcomputer.id = db_dbauthinfo.dbcomputer_id
Info: DbComputer.authinfos setup secondary join None
Info: DbComputer.authinfos synchronize pairs [(db_dbcomputer.id => db_dbauthinfo.dbcomputer_id)]
Info: DbComputer.authinfos secondary synchronize pairs []
Info: DbComputer.authinfos local/remote pairs [(db_dbcomputer.id / db_dbauthinfo.dbcomputer_id)]
Info: DbComputer.authinfos remote columns [db_dbauthinfo.dbcomputer_id]
Info: DbComputer.authinfos local columns [db_dbcomputer.id]
Info: DbComputer.authinfos relationship direction symbol('ONETOMANY')
Info: DbComputer.authinfos lazy loading clause :param_1 = db_dbauthinfo.dbcomputer_id
Info: DbAuthInfo.dbcomputer lazy loading clause db_dbcomputer.id = :param_1
Info: DbAuthInfo.dbcomputer will use Session.get() to optimize instance loads
Info: (DbAuthInfo|db_dbauthinfo) initialize prop id
Info: (DbAuthInfo|db_dbauthinfo) initialize prop aiidauser_id
Info: (DbAuthInfo|db_dbauthinfo) initialize prop dbcomputer_id
Info: (DbAuthInfo|db_dbauthinfo) initialize prop auth_params
Info: (DbAuthInfo|db_dbauthinfo) initialize prop enabled
Info: (DbAuthInfo|db_dbauthinfo) _post_configure_properties() complete
Info: (DbComputer|db_dbcomputer) _post_configure_properties() started
Info: (DbComputer|db_dbcomputer) initialize prop _metadata
Info: (DbComputer|db_dbcomputer) initialize prop id
Info: (DbComputer|db_dbcomputer) initialize prop uuid
Info: (DbComputer|db_dbcomputer) initialize prop label
Info: (DbComputer|db_dbcomputer) initialize prop hostname
Info: (DbComputer|db_dbcomputer) initialize prop description
Info: (DbComputer|db_dbcomputer) initialize prop scheduler_type
Info: (DbComputer|db_dbcomputer) initialize prop transport_type
Info: (DbComputer|db_dbcomputer) initialize prop authinfos
Info: (DbComputer|db_dbcomputer) _post_configure_properties() complete
Info: (DbUser|db_dbuser) _post_configure_properties() started
Info: (DbUser|db_dbuser) initialize prop id
Info: (DbUser|db_dbuser) initialize prop email
Info: (DbUser|db_dbuser) initialize prop first_name
Info: (DbUser|db_dbuser) initialize prop last_name
Info: (DbUser|db_dbuser) initialize prop institution
Info: (DbUser|db_dbuser) initialize prop authinfos
Info: (DbUser|db_dbuser) _post_configure_properties() complete
Info: (DbComment|db_dbcomment) _post_configure_properties() started
Info: (DbComment|db_dbcomment) initialize prop dbnode
Info: DbComment.dbnode setup primary join db_dbnode.id = db_dbcomment.dbnode_id
Info: DbComment.dbnode setup secondary join None
Info: DbComment.dbnode synchronize pairs [(db_dbnode.id => db_dbcomment.dbnode_id)]
Info: DbComment.dbnode secondary synchronize pairs []
Info: DbComment.dbnode local/remote pairs [(db_dbcomment.dbnode_id / db_dbnode.id)]
Info: DbComment.dbnode remote columns [db_dbnode.id]
Info: DbComment.dbnode local columns [db_dbcomment.dbnode_id]
Info: DbComment.dbnode relationship direction symbol('MANYTOONE')
Info: (DbNode|db_dbnode) _configure_property(dbcomments, RelationshipProperty)
Info: DbNode.dbcomments setup primary join db_dbnode.id = db_dbcomment.dbnode_id
Info: DbNode.dbcomments setup secondary join None
Info: DbNode.dbcomments synchronize pairs [(db_dbnode.id => db_dbcomment.dbnode_id)]
Info: DbNode.dbcomments secondary synchronize pairs []
Info: DbNode.dbcomments local/remote pairs [(db_dbnode.id / db_dbcomment.dbnode_id)]
Info: DbNode.dbcomments remote columns [db_dbcomment.dbnode_id]
Info: DbNode.dbcomments local columns [db_dbnode.id]
Info: DbNode.dbcomments relationship direction symbol('ONETOMANY')
Info: DbNode.dbcomments lazy loading clause :param_1 = db_dbcomment.dbnode_id
Info: DbComment.dbnode lazy loading clause db_dbnode.id = :param_1
Info: DbComment.dbnode will use Session.get() to optimize instance loads
Info: (DbComment|db_dbcomment) initialize prop user
Info: DbComment.user setup primary join db_dbuser.id = db_dbcomment.user_id
Info: DbComment.user setup secondary join None
Info: DbComment.user synchronize pairs [(db_dbuser.id => db_dbcomment.user_id)]
Info: DbComment.user secondary synchronize pairs []
Info: DbComment.user local/remote pairs [(db_dbcomment.user_id / db_dbuser.id)]
Info: DbComment.user remote columns [db_dbuser.id]
Info: DbComment.user local columns [db_dbcomment.user_id]
Info: DbComment.user relationship direction symbol('MANYTOONE')
Info: DbComment.user lazy loading clause db_dbuser.id = :param_1
Info: DbComment.user will use Session.get() to optimize instance loads
Info: (DbComment|db_dbcomment) initialize prop id
Info: (DbComment|db_dbcomment) initialize prop uuid
Info: (DbComment|db_dbcomment) initialize prop dbnode_id
Info: (DbComment|db_dbcomment) initialize prop ctime
Info: (DbComment|db_dbcomment) initialize prop mtime
Info: (DbComment|db_dbcomment) initialize prop user_id
Info: (DbComment|db_dbcomment) initialize prop content
Info: (DbComment|db_dbcomment) _post_configure_properties() complete
Info: (DbGroupNode|db_dbgroup_dbnodes) _post_configure_properties() started
Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop id
Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop dbnode_id
Info: (DbGroupNode|db_dbgroup_dbnodes) initialize prop dbgroup_id
Info: (DbGroupNode|db_dbgroup_dbnodes) _post_configure_properties() complete
Info: (DbGroup|db_dbgroup) _post_configure_properties() started
Info: (DbGroup|db_dbgroup) initialize prop user
Info: DbGroup.user setup primary join db_dbuser.id = db_dbgroup.user_id
Info: DbGroup.user setup secondary join None
Info: DbGroup.user synchronize pairs [(db_dbuser.id => db_dbgroup.user_id)]
Info: DbGroup.user secondary synchronize pairs []
Info: DbGroup.user local/remote pairs [(db_dbgroup.user_id / db_dbuser.id)]
Info: DbGroup.user remote columns [db_dbuser.id]
Info: DbGroup.user local columns [db_dbgroup.user_id]
Info: DbGroup.user relationship direction symbol('MANYTOONE')
Info: (DbUser|db_dbuser) _configure_property(dbgroups, RelationshipProperty)
Info: DbUser.dbgroups setup primary join db_dbuser.id = db_dbgroup.user_id
Info: DbUser.dbgroups setup secondary join None
Info: DbUser.dbgroups synchronize pairs [(db_dbuser.id => db_dbgroup.user_id)]
Info: DbUser.dbgroups secondary synchronize pairs []
Info: DbUser.dbgroups local/remote pairs [(db_dbuser.id / db_dbgroup.user_id)]
Info: DbUser.dbgroups remote columns [db_dbgroup.user_id]
Info: DbUser.dbgroups local columns [db_dbuser.id]
Info: DbUser.dbgroups relationship direction symbol('ONETOMANY')
Info: DbUser.dbgroups lazy loading clause :param_1 = db_dbgroup.user_id
Info: DbGroup.user lazy loading clause db_dbuser.id = :param_1
Info: DbGroup.user will use Session.get() to optimize instance loads
Info: (DbGroup|db_dbgroup) initialize prop dbnodes
Info: DbGroup.dbnodes setup primary join db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id
Info: DbGroup.dbnodes setup secondary join db_dbnode.id = db_dbgroup_dbnodes.dbnode_id
Info: DbGroup.dbnodes synchronize pairs [(db_dbgroup.id => db_dbgroup_dbnodes.dbgroup_id)]
Info: DbGroup.dbnodes secondary synchronize pairs [(db_dbnode.id => db_dbgroup_dbnodes.dbnode_id)]
Info: DbGroup.dbnodes local/remote pairs [(db_dbgroup.id / db_dbgroup_dbnodes.dbgroup_id),(db_dbnode.id / db_dbgroup_dbnodes.dbnode_id)]
Info: DbGroup.dbnodes remote columns [db_dbgroup_dbnodes.dbgroup_id,db_dbgroup_dbnodes.dbnode_id]
Info: DbGroup.dbnodes local columns [db_dbgroup.id]
Info: DbGroup.dbnodes relationship direction symbol('MANYTOMANY')
Info: (DbNode|db_dbnode) _configure_property(dbgroups, RelationshipProperty)
Info: DbNode.dbgroups setup primary join db_dbnode.id = db_dbgroup_dbnodes.dbnode_id
Info: DbNode.dbgroups setup secondary join db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id
Info: DbNode.dbgroups synchronize pairs [(db_dbnode.id => db_dbgroup_dbnodes.dbnode_id)]
Info: DbNode.dbgroups secondary synchronize pairs [(db_dbgroup.id => db_dbgroup_dbnodes.dbgroup_id)]
Info: DbNode.dbgroups local/remote pairs [(db_dbnode.id / db_dbgroup_dbnodes.dbnode_id),(db_dbgroup.id / db_dbgroup_dbnodes.dbgroup_id)]
Info: DbNode.dbgroups remote columns [db_dbgroup_dbnodes.dbgroup_id,db_dbgroup_dbnodes.dbnode_id]
Info: DbNode.dbgroups local columns [db_dbnode.id]
Info: DbNode.dbgroups relationship direction symbol('MANYTOMANY')
Info: DbNode.dbgroups lazy loading clause :param_1 = db_dbgroup_dbnodes.dbnode_id AND db_dbgroup.id = db_dbgroup_dbnodes.dbgroup_id
Info: DbGroup.dbnodes lazy loading clause :param_1 = db_dbgroup_dbnodes.dbgroup_id AND db_dbnode.id = db_dbgroup_dbnodes.dbnode_id
Info: (DbGroup|db_dbgroup) initialize prop id
Info: (DbGroup|db_dbgroup) initialize prop uuid
Info: (DbGroup|db_dbgroup) initialize prop label
Info: (DbGroup|db_dbgroup) initialize prop type_string
Info: (DbGroup|db_dbgroup) initialize prop time
Info: (DbGroup|db_dbgroup) initialize prop description
Info: (DbGroup|db_dbgroup) initialize prop extras
Info: (DbGroup|db_dbgroup) initialize prop user_id
Info: (DbGroup|db_dbgroup) _post_configure_properties() complete
Info: (DbLog|db_dblog) _post_configure_properties() started
Info: (DbLog|db_dblog) initialize prop _metadata
Info: (DbLog|db_dblog) initialize prop dbnode
Info: DbLog.dbnode setup primary join db_dbnode.id = db_dblog.dbnode_id
Info: DbLog.dbnode setup secondary join None
Info: DbLog.dbnode synchronize pairs [(db_dbnode.id => db_dblog.dbnode_id)]
Info: DbLog.dbnode secondary synchronize pairs []
Info: DbLog.dbnode local/remote pairs [(db_dblog.dbnode_id / db_dbnode.id)]
Info: DbLog.dbnode remote columns [db_dbnode.id]
Info: DbLog.dbnode local columns [db_dblog.dbnode_id]
Info: DbLog.dbnode relationship direction symbol('MANYTOONE')
Info: (DbNode|db_dbnode) _configure_property(dblogs, RelationshipProperty)
Info: DbNode.dblogs setup primary join db_dbnode.id = db_dblog.dbnode_id
Info: DbNode.dblogs setup secondary join None
Info: DbNode.dblogs synchronize pairs [(db_dbnode.id => db_dblog.dbnode_id)]
Info: DbNode.dblogs secondary synchronize pairs []
Info: DbNode.dblogs local/remote pairs [(db_dbnode.id / db_dblog.dbnode_id)]
Info: DbNode.dblogs remote columns [db_dblog.dbnode_id]
Info: DbNode.dblogs local columns [db_dbnode.id]
Info: DbNode.dblogs relationship direction symbol('ONETOMANY')
Info: DbNode.dblogs lazy loading clause :param_1 = db_dblog.dbnode_id
Info: DbLog.dbnode lazy loading clause db_dbnode.id = :param_1
Info: DbLog.dbnode will use Session.get() to optimize instance loads
Info: (DbLog|db_dblog) initialize prop id
Info: (DbLog|db_dblog) initialize prop uuid
Info: (DbLog|db_dblog) initialize prop time
Info: (DbLog|db_dblog) initialize prop loggername
Info: (DbLog|db_dblog) initialize prop levelname
Info: (DbLog|db_dblog) initialize prop dbnode_id
Info: (DbLog|db_dblog) initialize prop message
Info: (DbLog|db_dblog) _post_configure_properties() complete
Info: (DbNode|db_dbnode) _post_configure_properties() started
Info: (DbNode|db_dbnode) initialize prop dbcomputer
Info: DbNode.dbcomputer setup primary join db_dbcomputer.id = db_dbnode.dbcomputer_id
Info: DbNode.dbcomputer setup secondary join None
Info: DbNode.dbcomputer synchronize pairs [(db_dbcomputer.id => db_dbnode.dbcomputer_id)]
Info: DbNode.dbcomputer secondary synchronize pairs []
Info: DbNode.dbcomputer local/remote pairs [(db_dbnode.dbcomputer_id / db_dbcomputer.id)]
Info: DbNode.dbcomputer remote columns [db_dbcomputer.id]
Info: DbNode.dbcomputer local columns [db_dbnode.dbcomputer_id]
Info: DbNode.dbcomputer relationship direction symbol('MANYTOONE')
Info: (DbComputer|db_dbcomputer) _configure_property(dbnodes, RelationshipProperty)
Info: DbComputer.dbnodes setup primary join db_dbcomputer.id = db_dbnode.dbcomputer_id
Info: DbComputer.dbnodes setup secondary join None
Info: DbComputer.dbnodes synchronize pairs [(db_dbcomputer.id => db_dbnode.dbcomputer_id)]
Info: DbComputer.dbnodes secondary synchronize pairs []
Info: DbComputer.dbnodes local/remote pairs [(db_dbcomputer.id / db_dbnode.dbcomputer_id)]
Info: DbComputer.dbnodes remote columns [db_dbnode.dbcomputer_id]
Info: DbComputer.dbnodes local columns [db_dbcomputer.id]
Info: DbComputer.dbnodes relationship direction symbol('ONETOMANY')
Info: DbComputer.dbnodes lazy loading clause :param_1 = db_dbnode.dbcomputer_id
Info: DbNode.dbcomputer lazy loading clause db_dbcomputer.id = :param_1
Info: DbNode.dbcomputer will use Session.get() to optimize instance loads
Info: (DbNode|db_dbnode) initialize prop user
Info: DbNode.user setup primary join db_dbuser.id = db_dbnode.user_id
Info: DbNode.user setup secondary join None
Info: DbNode.user synchronize pairs [(db_dbuser.id => db_dbnode.user_id)]
Info: DbNode.user secondary synchronize pairs []
Info: DbNode.user local/remote pairs [(db_dbnode.user_id / db_dbuser.id)]
Info: DbNode.user remote columns [db_dbuser.id]
Info: DbNode.user local columns [db_dbnode.user_id]
Info: DbNode.user relationship direction symbol('MANYTOONE')
Info: (DbUser|db_dbuser) _configure_property(dbnodes, RelationshipProperty)
Info: DbUser.dbnodes setup primary join db_dbuser.id = db_dbnode.user_id
Info: DbUser.dbnodes setup secondary join None
Info: DbUser.dbnodes synchronize pairs [(db_dbuser.id => db_dbnode.user_id)]
Info: DbUser.dbnodes secondary synchronize pairs []
Info: DbUser.dbnodes local/remote pairs [(db_dbuser.id / db_dbnode.user_id)]
Info: DbUser.dbnodes remote columns [db_dbnode.user_id]
Info: DbUser.dbnodes local columns [db_dbuser.id]
Info: DbUser.dbnodes relationship direction symbol('ONETOMANY')
Info: DbUser.dbnodes lazy loading clause :param_1 = db_dbnode.user_id
Info: DbNode.user lazy loading clause db_dbuser.id = :param_1
Info: DbNode.user will use Session.get() to optimize instance loads
Info: (DbNode|db_dbnode) initialize prop outputs_q
Info: DbNode.outputs_q setup primary join db_dbnode.id = db_dblink.input_id
Info: DbNode.outputs_q setup secondary join db_dbnode.id = db_dblink.output_id
Info: DbNode.outputs_q synchronize pairs [(db_dbnode.id => db_dblink.input_id)]
Info: DbNode.outputs_q secondary synchronize pairs [(db_dbnode.id => db_dblink.output_id)]
Info: DbNode.outputs_q local/remote pairs [(db_dbnode.id / db_dblink.input_id),(db_dbnode.id / db_dblink.output_id)]
Info: DbNode.outputs_q remote columns [db_dblink.input_id,db_dblink.output_id]
Info: DbNode.outputs_q local columns [db_dbnode.id]
Info: DbNode.outputs_q relationship direction symbol('MANYTOMANY')
Info: (DbNode|db_dbnode) _configure_property(inputs_q, RelationshipProperty)
Info: DbNode.inputs_q setup primary join db_dbnode.id = db_dblink.output_id
Info: DbNode.inputs_q setup secondary join db_dbnode.id = db_dblink.input_id
Info: DbNode.inputs_q synchronize pairs [(db_dbnode.id => db_dblink.output_id)]
Info: DbNode.inputs_q secondary synchronize pairs [(db_dbnode.id => db_dblink.input_id)]
Info: DbNode.inputs_q local/remote pairs [(db_dbnode.id / db_dblink.output_id),(db_dbnode.id / db_dblink.input_id)]
Info: DbNode.inputs_q remote columns [db_dblink.input_id,db_dblink.output_id]
Info: DbNode.inputs_q local columns [db_dbnode.id]
Info: DbNode.inputs_q relationship direction symbol('MANYTOMANY')
Info: DbNode.inputs_q lazy loading clause :param_1 = db_dblink.output_id AND db_dbnode.id = db_dblink.input_id
Info: DbNode.outputs_q lazy loading clause :param_1 = db_dblink.input_id AND db_dbnode.id = db_dblink.output_id
Info: (DbNode|db_dbnode) initialize prop id
Info: (DbNode|db_dbnode) initialize prop uuid
Info: (DbNode|db_dbnode) initialize prop node_type
Info: (DbNode|db_dbnode) initialize prop process_type
Info: (DbNode|db_dbnode) initialize prop label
Info: (DbNode|db_dbnode) initialize prop description
Info: (DbNode|db_dbnode) initialize prop ctime
Info: (DbNode|db_dbnode) initialize prop mtime
Info: (DbNode|db_dbnode) initialize prop attributes
Info: (DbNode|db_dbnode) initialize prop extras
Info: (DbNode|db_dbnode) initialize prop repository_metadata
Info: (DbNode|db_dbnode) initialize prop dbcomputer_id
Info: (DbNode|db_dbnode) initialize prop user_id
Info: (DbNode|db_dbnode) initialize prop dbcomments
Info: (DbNode|db_dbnode) initialize prop dbgroups
Info: (DbNode|db_dbnode) initialize prop dblogs
Info: (DbNode|db_dbnode) _post_configure_properties() complete
Info: (DbLink|db_dblink) _post_configure_properties() started
Info: (DbLink|db_dblink) initialize prop input
Info: DbLink.input setup primary join db_dblink.input_id = db_dbnode.id
Info: DbLink.input setup secondary join None
Info: DbLink.input synchronize pairs [(db_dbnode.id => db_dblink.input_id)]
Info: DbLink.input secondary synchronize pairs []
Info: DbLink.input local/remote pairs [(db_dblink.input_id / db_dbnode.id)]
Info: DbLink.input remote columns [db_dbnode.id]
Info: DbLink.input local columns [db_dblink.input_id]
Info: DbLink.input relationship direction symbol('MANYTOONE')
Info: DbLink.input lazy loading clause :param_1 = db_dbnode.id
Info: DbLink.input will use Session.get() to optimize instance loads
Info: (DbLink|db_dblink) initialize prop output
Info: DbLink.output setup primary join db_dblink.output_id = db_dbnode.id
Info: DbLink.output setup secondary join None
Info: DbLink.output synchronize pairs [(db_dbnode.id => db_dblink.output_id)]
Info: DbLink.output secondary synchronize pairs []
Info: DbLink.output local/remote pairs [(db_dblink.output_id / db_dbnode.id)]
Info: DbLink.output remote columns [db_dbnode.id]
Info: DbLink.output local columns [db_dblink.output_id]
Info: DbLink.output relationship direction symbol('MANYTOONE')
Info: DbLink.output lazy loading clause :param_1 = db_dbnode.id
Info: DbLink.output will use Session.get() to optimize instance loads
Info: (DbLink|db_dblink) initialize prop id
Info: (DbLink|db_dblink) initialize prop input_id
Info: (DbLink|db_dblink) initialize prop output_id
Info: (DbLink|db_dblink) initialize prop label
Info: (DbLink|db_dblink) initialize prop type
Info: (DbLink|db_dblink) _post_configure_properties() complete
Info: SELECT db_dbsetting.val 
FROM db_dbsetting 
WHERE db_dbsetting.key = %(key_1)s
Info: [generated in 0.00007s] {'key_1': 'repository|uuid'}
Info: ROLLBACK
Info: Pool disposed. Pool size: 5  Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0
Info: Pool recreating
Info: select pg_catalog.version()
Info: [raw sql] {}
Info: select current_schema()
Info: [raw sql] {}
Info: show standard_conforming_strings
Info: [raw sql] {}
Info: BEGIN (implicit)
Info: SAVEPOINT sa_savepoint_1
Info: [no key 0.00005s] {}
Info: SELECT db_dbnode_1.id, db_dbnode_1.uuid, db_dbnode_1.node_type, db_dbnode_1.process_type, db_dbnode_1.label, db_dbnode_1.description, db_dbnode_1.ctime, db_dbnode_1.mtime, db_dbnode_1.attributes, db_dbnode_1.extras, db_dbnode_1.repository_metadata, db_dbnode_1.dbcomputer_id, db_dbnode_1.user_id 
FROM db_dbnode AS db_dbnode_1 
WHERE CAST(db_dbnode_1.node_type AS VARCHAR) LIKE %(param_1)s AND db_dbnode_1.id = %(id_1)s 
 LIMIT %(param_2)s
Info: [generated in 0.00011s] {'param_1': 'process.%', 'id_1': 12985, 'param_2': 2}
Info: RELEASE SAVEPOINT sa_savepoint_1
Info: [no key 0.00005s] {}
Info: SELECT db_dbnode.attributes AS db_dbnode_attributes 
FROM db_dbnode 
WHERE db_dbnode.id = %(pk_1)s
Info: [generated in 0.00010s] {'pk_1': 12985}
Info: SELECT db_dbnode.attributes AS db_dbnode_attributes 
FROM db_dbnode 
WHERE db_dbnode.id = %(pk_1)s
Info: [cached since 0.00113s ago] {'pk_1': 12985}
Info: SELECT db_dbnode.attributes AS db_dbnode_attributes 
FROM db_dbnode 
WHERE db_dbnode.id = %(pk_1)s
Info: [cached since 0.001908s ago] {'pk_1': 12985}
Error: Process<12985> is unreachable.

[sphuber]

Yeah, but that is probably because you ran verdi devel rabbitmq tasks revive 12985 even though it wasn't a zombie, otherwise the verdi devel rabbitmq tasks analyze would have noticed. Please run verdi devel rabbitmq tasks analyze again to make sure everything is consistent there. The fact that it is not reachable, most likely means the task got lost.

[mbercx]

Just a few comments on the UX here:

1. Playing the same process multiple times returns the same report. Is it possible to print a message like "Critical: the process is already running" instead?

   ❯ verdi process play 18189
   Report: played Process<18189>
   ❯ verdi process play 18189
   Report: played Process<18189>
   

2. The verdi process play/pause commands can sometimes seemingly hang. As the impatient bugger I am, I then abort them, only to find out that the process has actually been played/paused:

   ❯ verdi process pause 16934
   ^C
   Aborted!
   ❯ verdi process list
      PK  Created    Process label      Process State    Process status
   -----  ---------  -----------------  ---------------  ---------------------------------------
   [...]
   16934  5D ago     SuperConWorkChain  ⏸ Waiting        Paused through `verdi process pause`
   [...]
   
   Total results: 11
   
   Report: last time an entry changed state: 4s ago (at 07:54:01 on 2023-06-05)
   Report: Checking daemon load... OK
   Report: Using 6% of the available daemon worker slots.
   

3. After the process above (16934) played, it started running another calculation again, but didn't show that it was waiting the whole time the calculation was running:

   ❯ verdi process list
      PK  Created    Process label      Process State    Process status
   -----  ---------  -----------------  ---------------  ---------------------------------------
   [...]
   16934  5D ago     SuperConWorkChain  ⏵ Waiting
   17009  5D ago     SuperConWorkChain  ⏵ Waiting        Waiting for child processes: 18238
   18189  15h ago    SuperConWorkChain  ⏵ Waiting        Waiting for child processes: 18230
   18230  5m ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state RUNNING
   18238  3m ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state RUNNING
   18246  1m ago     EpwCalculation     ⏵ Waiting        Monitoring scheduler: job state UNKNOWN
   
   Total results: 11
   
   Report: last time an entry changed state: 15s ago (at 07:55:13 on 2023-06-05)
   Report: Checking daemon load... OK
   Report: Using 6% of the available daemon worker slots.
   ❯ verdi process call-root 18246
   16934
   

[sphuber]

Before answering your explicit points, I first want to explain that in general it is going to be difficult to make the behavior of these RPC commands fully intuitive. A user may expect that if you call verdi process play that it takes immediate effect. However, that is not the case. What is actually done is that a remote procedure call is sent over RabbitMQ. This means it sends a request a command to be executed on a specific process isntance by the daemon worker that is currently working on that process. This command will have to enter on the event loop just as all the other things that the worker is already doing. If the worker is busy, it may not get to it immediately and so the command will not be executed immediately. There will therefore typically be a delay between the RPC having been sent, and it actually be executed and the effect visible.

It is not going to be possible to "easily" explain this to users and convey this in the output of the commands. This PR should maybe help though: #6048
It will essentially change the language from:

played Process<ID>

to

request to play Process<ID> sent

Hopefully this should indicate that it is not a direct action, but something that will have to be resolved.

Playing the same process multiple times returns the same report. Is it possible to print a message like "Critical: the process is already running" instead?

See above: we could check the paused attribute on the node, and if it doesn't exist, raise the error, but as noted there can be (temporary) problems with the Process instance and the ProcessNode proxy being out of sync. If we were to change this, I would maybe just add a warning instead. When play is called, checked whether ProcessNode.paused is True and if so emit a warning saying Process<ID> should already be playing.

The verdi process play/pause commands can sometimes seemingly hang. As the impatient bugger I am, I then abort them, only to find out that the process has actually been played/paused:

Not much we can do here I think. What I suspect is happening is that there is a lot of traffic and so the response of the RPC being delivered is taking a bit longer, but at some point it did get delivered, but before the response actually got back to you, you killed the command. So the action will be performed eventually.

After the process above (16934) played, it started running another calculation again, but didn't show that it was waiting the whole time the calculation was running

I think this is due to the hard shutdown of the daemon. The Process status is populated based on the process_status attribute of the node. If a node is paused, it is said in human-readable from in that attribute, overwriting anything that was there. This should be restored once it is played again, but perhaps this didn't happen in this case? There are a lot of possible pathways, so difficult to say with certainty. But this could potentially be fixed.