feat: composite queries (#4003)

Add new form of composite query that:

- can call other queries and composite queries
- can't be called from updates
- can't do async expressions (no way to queue up a callback since state is lost)
- can't call local async or async* (because those might contain update calls)

From the (informal) doc:

# Composite query functions

Although queries can be fast, when called from a frontend, yet trusted though slower, when called from an actor, they are also limited in what they can do. In particular, they cannot themselves issue further messages, including queries.

To address this limitation, the Internet Computer supports another flavour of query function called a composite query. Like plain queries, the state changes made by a composite query are transient, isolated and never committed. Moreover, composite queries cannot call update functions, including those implicit in async expressions (which require update calls under the hood). Unlike plain queries, composite queries can call query functions and composite query functions, on the same and other actors, but only provided those actors reside on the same subnet.

As a contrived example, consider generalising the previous Counter actor to a class of counters. Each instance of the class provides an additional composite query to sum the values of a given array of counters:
```
actor class Counter () {

  var count = 0;

  // ...

  public shared query func peek() : async Nat {
    count
  };

  public shared composite query func sum(counters : [Counter]) : async Nat {
    var sum = 0;
    for (counter in counters.vals())  {
      sum += await counter.peek();
    };
    sum
  }

}
```

Declaring `sum` as a composite query enables it call the peek queries of its argument counters.

While update message can call plain query functions, they cannot call composite query functions. This distinction, which is dictated by the current capabilites of the IC, explains why query functions and composite query functions are regarded as distinct types of shared functions.

Note that the composite query modifier is reflected in the type of a composite query function:
```
  sum : shared composite query ([Counter]) -> async Nat
```

Since only a composite query can call another composite query, you may be wondering how any composite query gets called at all? The answer to this chicken-and-egg problem is that composite queries are initiated outside the IC, typically by an application (such as a browser frontend) sending an ingress message invoking a composite query on a backend actor on the IC.



c.f. dfinity/interface-spec#144

- [X] positive tests
- [X] negative tests
- [X] RTS support (subtyping, IDL validation etc)
- [x] doc
- [x] gc (to GC or not at end of each message)? We currently do a (redundant?) GC for composite queries
- [x] imports
- [x] inspect_message support? Since composite queries can't be called as updates, perhaps they don't matter at all?
- [x] I had to modify cycles adding and refund logic to avoid calling ic0.call_cycles_add128/ic0.cycles_refunded128 unless necessary - these aren't supported for composite query callbacks (see spec).
- [x] More accurate Lifetime modelling - we currently reuse state Lifecycle.InUpdate since callbacks are always in this state (and changeing that statically is tricky without introducing new forms of callback for CompositeQueries. Adding a new state would allow us to suppress GC at the end of each message, if desired, and more elegantly rule out forbidden ops like adding non-zero cycles, observing refunds. Of course, we could also just set a global to disable these until all state-changes are aborted at the end. Might be simpler.
- [x] regenerate ok test output once ic-ref available again by merge with master. At the moment, I can manually run the new tests, but can't accept their output. They pass, trust me.
- [x] better error messages, distinguishing capability  (send update, send query, send composite query, perhaps) required.
- [ ] enable ic-ref tests (once ic-ref supports composite queries)
- [x] add keyword `composite` to idl, local highlight.js

Changelog.md

@@ -2,13 +2,40 @@
 
 * motoko (`moc`)
 
+  * BREAKING CHANGE (Minor):
+
+    New keyword `composite` allows one to declare Internet Computer *composite queries* (#4003).
+
+    For example,
+
+    ``` motoko
+    public shared composite query func sum(counters : [Counter]) : async Nat {
+      var sum = 0;
+      for (counter in counters.vals())  {
+        sum += await counter.peek();
+      };
+      sum
+    }
+    ```
+
+    has type:
+
+    ``` motoko
+    shared composite query [Counter] -> async Nat
+    ```
+
+    and can call both `query` and other `composite query` functions.
+
+    See the documentation for full details.
+
   * Allow canister imports of Candid service constructors, ignoring the service arguments to
     import the instantiated service instead (with a warning) (#4041).
 
   * Allow optional terminal semicolons in Candid imports (#4042).
 
   * bugfix: allow signed float literals as static expressions in modules (#4063).
 
+
 ## 0.9.3 (2023-06-19)
 
 * motoko (`moc`)

doc/docusaurus/src/theme/CodeBlock/hljs_run.js

@@ -78,7 +78,7 @@ export function registerMotoko() {
         $pattern: "[a-zA-Z_]\\w*",
         keyword:
           "actor and await break case catch class" +
-          " continue debug do else for func if in import" +
+          " continue composite debug do else for func if in import" +
           " module not object or label let loop private" +
           " public return shared try throw query switch" +
           " type var while with stable flexible system debug_show assert ignore from_candid to_candid",

doc/md/actors-async.md

@@ -184,7 +184,8 @@ The `query` modifier is reflected in the type of a query function:
 
 As before, in `query` declarations and actor types the `shared` keyword can be omitted.
 
-## Messaging Restrictions
+
+## Messaging restrictions
 
 The Internet Computer places restrictions on when and how canisters are allowed to communicate. These restrictions are enforced dynamically on the Internet Computer but prevented statically in Motoko, ruling out a class of dynamic execution errors. Two examples are:
 
@@ -196,7 +197,7 @@ These restrictions are surfaced in Motoko as restrictions on the context in whic
 
 In Motoko, an expression occurs in an *asynchronous context* if it appears in the body of an `async` expression, which may be the body of a (shared or local) function or a stand-alone expression. The only exception are `query` functions, whose body is not considered to open an asynchronous context.
 
-In Motoko calling a shared function is an error unless the function is called in an asynchronouus context. In addition, calling a shared function from an actor class constructor is also an error.
+In Motoko calling a shared function is an error unless the function is called in an asynchronous context. In addition, calling a shared function from an actor class constructor is also an error.
 
 The `await` construct is only allowed in an asynchronous context.
 
@@ -238,3 +239,68 @@ The last two lines above *instantiate* the actor class twice. The first invocati
 For now, the Motoko compiler gives an error when compiling programs that do not consist of a single actor or actor class. Compiled programs may still, however, reference imported actor classes. For more information, see [Importing actor classes](modules-and-imports.md#importing-actor-classes) and [Actor classes](actor-classes.md#actor-classes).
 
 :::
+
+## Composite query functions
+
+Although queries can be fast, when called from a frontend, yet trusted though slower, when called from an actor, they are also limited in what they can do.
+In particular, they cannot themselves issue further messages, including queries.
+
+To address this limitation, the Internet Computer supports another flavour of query function called a *composite query*.
+Like plain queries, the state changes made by a composite query are transient, isolated and never committed. Moreover, composite queries cannot call update functions, including those
+implicit in `async` expressions (which require update calls under the hood).
+Unlike plain queries, composite queries can call query functions and composite query functions, on the same and other actors, but only provided those actors reside on the same subnet.
+
+As a contrived example, consider generalising the previous `Counter` actor to a class of counters.
+Each instance of the class provides an additional `composite query` to sum the values
+of a given array of counters:
+
+``` motoko file=./examples/CounterWithCompositeQuery.mo
+```
+
+Declaring `sum` as a `composite query` enables it call the `peek` queries of its argument counters.
+
+While *update* message can call plain query functions, they cannot call *composite* query functions.
+This distinction, which is dictated by the current capabilites of the IC,
+explains why query functions and composite query functions are regarded as distinct types of shared functions.
+
+Note that the `composite query` modifier is reflected in the type of a composite query function:
+
+``` motoko no-repl
+  sum : shared composite query ([Counter]) -> async Nat
+```
+
+Since only a composite query can call another composite query, you may be wondering how any composite query gets called at all?
+The answer to this chicken-and-egg problem is that composite queries are initiated
+outside the IC, typically by an application (such as a browser
+frontend) sending an ingress message invoking a composite query on a backend
+actor on the IC.
+
+:::danger
+
+The Internet Computer's semantics of composite queries, like queries, ensures that state changes made by a composite query are isolated from other inter-canister calls,
+including recursive queries, composite or not, to the same actor.
+
+In particular, like a query, a composite query call rolls back its state on function exit, but is also does not pass state changes to sub-query or sub-composite-query calls.
+Therefore, repeated calls (which includes recursive calls) have different semantics from the more familiar sequential calls that accumulate state changes.
+
+In sequential calls to queries made by a composite query, the internal state changes of preceeding queries will have no effect on subsequent queries, nor will
+the queries observe any local state changes made by the enclosing composite query.
+Local states changes made by the composite query are, however, preserved across the calls until finally being rolled-back on exit from the composite query.
+
+This semantics can lead to surprising behaviour for users accustomed to ordinary imperative programming.
+
+Consider this contrived example containing the composite query `test` that calls query `q` and composite query `cq`.
+
+
+``` motoko no-repl file=./examples/CompositeSemantics.mo
+```
+
+When `state` is `0`, a call to `test` returns
+
+```
+{s0 = 0; s1 = 0; s2 = 0; s3 = 3_000}
+```
+
+because none of the local updates to `state` are visible to any of the callers or callees.
+
+:::

doc/md/examples/CompositeSemantics.mo

@@ -0,0 +1,31 @@
+actor Composites {
+
+  var state = 0;
+
+  // ...
+
+  public shared query func q() : async Nat {
+    let s = state;
+    state += 10;
+    s
+  };
+
+  public shared composite query func cq() : async Nat {
+    let s = state;
+    state += 100;
+    s
+  };
+
+  public shared composite query func test() :
+    async {s0 : Nat; s1 : Nat; s2 : Nat; s3 : Nat } {
+    let s0 = state;
+    state += 1000;
+    let s1 = await q();
+    state += 1000;
+    let s2 = await cq();
+    state += 1000;
+    let s3 = state;
+    {s0; s1; s2; s3}
+  };
+
+}

doc/md/examples/CounterWithCompositeQuery.mo

@@ -0,0 +1,19 @@
+actor class Counter () {
+
+  var count = 0;
+
+  // ...
+
+  public shared query func peek() : async Nat {
+    count
+  };
+
+  public shared composite query func sum(counters : [Counter]) : async Nat {
+    var sum = 0;
+    for (counter in counters.vals())  {
+      sum += await counter.peek();
+    };
+    sum
+  }
+
+}

doc/md/examples/grammar.txt

@@ -12,15 +12,19 @@
     'actor'
     'module'
 
+<query> ::= 
+    'query'
+    'composite' 'query'
+
 <func_sort_opt> ::= 
     <empty>
-    'shared' 'query'?
-    'query'
+    'shared' <query>?
+    <query>
 
 <shared_pat_opt> ::= 
     <empty>
-    'shared' 'query'? <pat_plain>?
-    'query' <pat_plain>?
+    'shared' <query>? <pat_plain>?
+    <query> <pat_plain>?
 
 <typ_obj> ::= 
     '{' <list(<typ_field>, ';')> '}'