experiment: deprecate experimental stable memory (improved) (#4575)

Leverages `/// @deprecate` comments to deprecate uses of ExperimentalStableMemory.

Adds moc flag `--experimental-stable-memory <n>` to control access to legacy stable memory:

- n<0: error on use of (ESM) prims.
- n=0: warn on use of prims.
- n=1: warning-less use of prims.

Current default is n=0 to encourage users to migrate their code to use preferred `Region.mo` without preventing compilation.

The implementation adds /// @deprecate M0199 comments Prim.mo and spots this specific deprecation during type-checking, reporting this specific violation just once per compilation unit, as an error, warning or not at all, depending on compiler setting.

The net effect is that, by default, any code that directly or indirectly imports ESM will be reported, as will any code that directly access a stable memory primitive, alerting the user to potential vulnerabilities. The code will still compile, but in future we can prevent that by defaulting n to 0, not 1.

The not-so-nice aspect of this implementation is that:
1: The errors only indicate that ESM *may* be in use. 
2: They do not pinpoint which importing library is responsible for the warning, but adding `/// @deprecate M0199` to corresponding ESM fields directly should achieve that too.

Changelog.md

@@ -4,6 +4,12 @@
 
 * motoko (`moc`)
 
+  * deprecation: Deprecate use of base library 'ExperimentalStableMemory' (ESM) (#4573).
+    New `moc` flag `--experimental-stable-memory <n>` controls the level of deprecation:
+    * n < 0: error on use of stable memory primitives.
+    * n = 0: warn on use of stable memory primitives.
+    * n > 1: warning-less use of stable memory primitives (for legacy applications).
+    Users of ESM should consider migrating their code to use isolated regions (library 'Region.mo') instead.
   * bugfix: Fix the detection of unused declarations in switch and catch cases (#4560).
   * improvement: Only warn on unused identifiers if type checking is error-free (#4560).
 

doc/md/reference/compiler-ref.md

@@ -36,6 +36,7 @@ You can use the following options with the `moc` command.
 | `--copying-gc`                            | Use copying GC (default).                                                                                                                             |
 | `--debug`                                 | Respects debug expressions in the source (the default).                                                                                               |
 | `--error-detail <n>`                      | Set level of error message detail for syntax errors, n in \[0..3\] (default 2).                                                                       |
+| `--experimental-stable-memory <n>`        | Select support for the deprecated `ExperimentalStableMemory.mo` library (n < 0: error, n = 0: warn, n > 0: allow) (default 0).                        |
 | `-fno-shared-code`                        | Do not share low-level utility code: larger code size but decreased cycle consumption (default).                                                      |
 | `-fshared-code`                           | Do share low-level utility code: smaller code size but increased cycle consumption.                                                                   |
 | `--generational-gc`                       | Use generational GC                                                                                                                                   |

src/exes/moc.ml

@@ -166,6 +166,10 @@ let argspec = [
   Arg.Unit (fun () -> Flags.force_gc := true),
   " disable GC scheduling, always do GC after an update message (for testing)";
 
+  "--experimental-stable-memory",
+  Arg.Set_int Flags.experimental_stable_memory,
+  " <n> select support for the deprecated `ExperimentalStableMemory.mo` library (n < 0: error, n == 0: warn, n > 0: allow) (default " ^ (Int.to_string Flags.experimental_stable_memory_default) ^ ")";
+
   "--max-stable-pages",
   Arg.Set_int Flags.max_stable_pages,
   "<n>  set maximum number of pages available for library `ExperimentalStableMemory.mo` (default " ^ (Int.to_string Flags.max_stable_pages_default) ^ ")";

src/lang_utils/error_codes.ml

@@ -202,4 +202,5 @@ let error_codes : (string * string option) list =
     "M0196", None; (* `system` capability supplied but not required *)
     "M0197", Some([%blob "lang_utils/error_codes/M0197.md"]); (* `system` capability required *)
     "M0198", Some([%blob "lang_utils/error_codes/M0198.md"]); (* Unused field pattern warning *)
+    "M0199", Some([%blob "lang_utils/error_codes/M0199.md"]); (* Deprecate experimental stable memory *)
   ]

src/lang_utils/error_codes/M0199.md

@@ -0,0 +1,19 @@
+# M0199
+
+This error or warning means that your code is either directly or indirectly using the now deprecated library `ExperimentalStableMemory.mo` (or its supporting compiler primitives).
+
+The library works as advertised but is a potential hazard as the resource it provides access to is shared between all clients of the library.
+This means that a library may unintentionally or maliciously read or modify data maintained by your application, or by another library imported by your application.
+
+If possible, please upgrade your code to use library `Region.mo` instead.
+This improved library offers a similar abstraction, but instead of a single memory that is implicitly accessible to all callers, it provides multiple memories.
+These memories, called regions, are isolated from each other and inaccessible unless a region is explicitly shared between libraries.
+
+The `moc` compiler flag `--experimental-stable-memory <n>` flag controls the production of this error or warning message, allowing your code to compile as before:
+* n < 0: error on use of stable memory primitives.
+* n = 0: warn on use of stable memory primitives (the default).
+* n > 1: warning-less use of stable memory primitives (for legacy applications).
+
+I.e. if your application cannot easily be upgraded to use `Regions.mo` and still requires access to `ExperimentalStableMemory.mo`, you can opt-in to legacy support for `ExperimentalStableMemory.mo` using the `moc` compiler flag `--experimental-stable-memory 1`.
+
+

src/mo_config/flags.ml

@@ -50,3 +50,5 @@ let rtti = ref false
 let trap_on_call_error = ref false
 let use_stable_regions = ref false
 let share_code = ref false
+let experimental_stable_memory_default = 0 (* _ < 0: error; _ = 0: warn, _ > 0: allow *)
+let experimental_stable_memory = ref experimental_stable_memory_default

src/mo_def/compUnit.ml

@@ -61,7 +61,12 @@ let comp_unit_of_prog as_lib (prog : prog) : comp_unit =
       if as_lib
       then
         (* Deprecated syntax, see Typing.check_lib *)
-        let fs = List.map (fun d -> {vis = Public None @@ no_region; dec = d; stab = None} @@ d.at) ds' in
+        (* Propagate deprecations *)
+        let fs = List.map (fun d ->
+          let trivia = Trivia.find_trivia prog.note.trivia d.at in
+          let depr = Trivia.deprecated_of_trivia_info trivia in
+          {vis = Public depr @@ no_region; dec = d; stab = None} @@ d.at) ds'
+        in
         finish imports {it = ModuleU (None, fs); at = no_region; note = empty_typ_note}
       else finish imports { it = ProgU ds; note = prog_typ_note; at = no_region }
   in

src/mo_frontend/typing.ml

@@ -55,6 +55,7 @@ type env =
     check_unused : bool;
     used_identifiers : S.t ref;
     unused_warnings : unused_warnings ref;
+    reported_stable_memory : bool ref;
   }
 
 let env_of_scope msgs scope =
@@ -76,6 +77,7 @@ let env_of_scope msgs scope =
     check_unused = true;
     used_identifiers = ref S.empty;
     unused_warnings = ref [];
+    reported_stable_memory = ref false;
   }
 
 let use_identifier env id =
@@ -146,6 +148,23 @@ let warn env at code fmt =
 let info env at fmt =
   Format.kasprintf (fun s -> Diag.add_msg env.msgs (type_info at s)) fmt
 
+let check_deprecation env at desc id depr =
+  match depr with
+  | Some ("M0199" as code) ->
+    if !(env.reported_stable_memory) then ()
+    else begin
+      env.reported_stable_memory := true;
+      (match compare !Flags.experimental_stable_memory 0 with
+       | -1 -> error
+       | 0 -> warn
+       | _ -> fun _ _ _ _ -> ())
+       env at code
+       "this code is (or uses) the deprecated library `ExperimentalStableMemory`.\nPlease use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message."
+    end
+  | Some msg ->
+    warn env at "M0154" "%s %s is deprecated:\n%s" desc id msg
+  | None -> ()
+
 let flag_of_compile_mode mode =
   match mode with
   | Flags.ICMode -> ""
@@ -407,9 +426,7 @@ and check_typ_path' env path : T.con =
     let s, fs = check_obj_path env path' in
     match T.lookup_typ_field id.it fs with
       | c ->
-        Option.iter
-          (warn env path.at "M0154" "type field %s is deprecated:\n%s" id.it)
-          (T.lookup_typ_deprecation id.it fs);
+        check_deprecation env path.at "type field" id.it (T.lookup_typ_deprecation id.it fs);
         c
       | exception Invalid_argument _ ->
         error env id.at "M0030" "type field %s does not exist in type%a"
@@ -1368,9 +1385,8 @@ and infer_exp'' env exp : T.typ =
         "cannot infer type of forward field reference %s"
         id.it
     | t ->
-      Option.iter
-        (warn env exp.at "M0154" "field %s is deprecated:\n%s" id.it)
-        (T.lookup_val_deprecation id.it tfs);
+      if not env.pre then
+        check_deprecation env exp.at "field" id.it (T.lookup_val_deprecation id.it tfs);
       t
     | exception Invalid_argument _ ->
       error env exp1.at "M0072"
@@ -2276,7 +2292,7 @@ and check_pat_fields env t tfs pfs ve at : Scope.val_env =
     | _ ->
       if T.is_mut typ then
         error env pf.at "M0120" "cannot pattern match mutable field %s" lab;
-      Option.iter (warn env pf.at "M0154" "type field %s is deprecated:\n%s" lab) src.T.depr;
+      check_deprecation env pf.at "field" lab src.T.depr;
       let val_kind = kind_of_field_pattern pf in
       let ve1 = check_pat_aux env typ pf.it.pat val_kind in
       let ve' =

src/pipeline/pipeline.ml

@@ -297,14 +297,18 @@ let prim_error phase (msgs : Diag.messages) =
 let check_prim () : Syntax.lib * stat_env =
   let lexer = Lexing.from_string (Prelude.prim_module ~timers:!Flags.global_timer) in
   let parse = Parser.Incremental.parse_prog in
-
   match parse_with Lexer.mode_priv lexer parse prim_name with
   | Error es -> prim_error "parsing" es
   | Ok (prog, _ws) ->
     let open Syntax in
     let open Source in
     let senv0 = initial_stat_env in
-    let fs = List.map (fun d -> {vis = Public None @@ no_region; dec = d; stab = None} @@ d.at) prog.it in
+    (* Propagate deprecations *)
+    let fs = List.map (fun d ->
+      let trivia = Trivia.find_trivia prog.note.trivia d.at in
+      let depr = Trivia.deprecated_of_trivia_info trivia in
+      {vis = Public depr @@ no_region; dec = d; stab = None} @@ d.at) prog.it
+    in
     let body = {it = ModuleU (None, fs); at = no_region; note = empty_typ_note} in
     let lib = {
       it = { imports = []; body };

src/prelude/internals.mo

@@ -17,7 +17,7 @@ func @add_cycles<system>() {
   let cycles = @cycles;
   @reset_cycles();
   if (cycles != 0) {
-    (prim "cyclesAdd" : <system>Nat -> ()) (cycles);
+    (prim "cyclesAdd" : Nat -> ()) (cycles);
   }
 };
 
@@ -416,7 +416,7 @@ func @install_actor_helper(
     switch install_arg {
       case (#new settings) {
         let available = (prim "cyclesAvailable" : () -> Nat) ();
-        let accepted = (prim "cyclesAccept" : <system>Nat -> Nat) (available);
+        let accepted = (prim "cyclesAccept" : Nat -> Nat) (available);
         let sender_canister_version = ?(prim "canister_version" : () -> Nat64)();
         @cycles += accepted;
         let { canister_id } =
@@ -449,7 +449,7 @@ func @install_actor_helper(
 // that Prim.createActor was mentioned on the forum and might be in use. (#3420)
 func @create_actor_helper(wasm_module : Blob, arg : Blob) : async Principal = async {
   let available = (prim "cyclesAvailable" : () -> Nat) ();
-  let accepted = (prim "cyclesAccept" : <system>Nat -> Nat) (available);
+  let accepted = (prim "cyclesAccept" : Nat -> Nat) (available);
   let sender_canister_version = ?(prim "canister_version" : () -> Nat64)();
   @cycles += accepted;
   let { canister_id } =

src/prelude/prim.mo

@@ -335,7 +335,7 @@ func cyclesRefunded() : Nat {
 };
 
 func cyclesAccept<system>(amount : Nat) : Nat {
-  (prim "cyclesAccept" : <system>Nat -> Nat)(amount);
+  (prim "cyclesAccept" : Nat -> Nat)(amount);
 };
 
 func cyclesAdd<system>(amount : Nat) : () {
@@ -353,48 +353,70 @@ func getCertificate() : ?Blob = (prim "getCertificate" : () -> ?Blob)();
 
 // stable memory
 
+/// @deprecated M0199
 func stableMemorySize() : Nat64 = (prim "stableMemorySize" : () -> Nat64)();
 
+/// @deprecated M0199
 func stableMemoryGrow(pages : Nat64) : Nat64 = (prim "stableMemoryGrow" : Nat64 -> Nat64) pages;
 
+/// @deprecated M0199
 func stableMemoryLoadNat32(offset : Nat64) : Nat32 = (prim "stableMemoryLoadNat32" : Nat64 -> Nat32) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreNat32(offset : Nat64, val : Nat32) : () = (prim "stableMemoryStoreNat32" : (Nat64, Nat32) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadNat8(offset : Nat64) : Nat8 = (prim "stableMemoryLoadNat8" : Nat64 -> Nat8) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreNat8(offset : Nat64, val : Nat8) : () = (prim "stableMemoryStoreNat8" : (Nat64, Nat8) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadNat16(offset : Nat64) : Nat16 = (prim "stableMemoryLoadNat16" : Nat64 -> Nat16) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreNat16(offset : Nat64, val : Nat16) : () = (prim "stableMemoryStoreNat16" : (Nat64, Nat16) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadNat64(offset : Nat64) : Nat64 = (prim "stableMemoryLoadNat64" : Nat64 -> Nat64) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreNat64(offset : Nat64, val : Nat64) : () = (prim "stableMemoryStoreNat64" : (Nat64, Nat64) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadInt32(offset : Nat64) : Int32 = (prim "stableMemoryLoadInt32" : Nat64 -> Int32) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreInt32(offset : Nat64, val : Int32) : () = (prim "stableMemoryStoreInt32" : (Nat64, Int32) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadInt8(offset : Nat64) : Int8 = (prim "stableMemoryLoadInt8" : Nat64 -> Int8) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreInt8(offset : Nat64, val : Int8) : () = (prim "stableMemoryStoreInt8" : (Nat64, Int8) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadInt16(offset : Nat64) : Int16 = (prim "stableMemoryLoadInt16" : Nat64 -> Int16) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreInt16(offset : Nat64, val : Int16) : () = (prim "stableMemoryStoreInt16" : (Nat64, Int16) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadInt64(offset : Nat64) : Int64 = (prim "stableMemoryLoadInt64" : Nat64 -> Int64) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreInt64(offset : Nat64, val : Int64) : () = (prim "stableMemoryStoreInt64" : (Nat64, Int64) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadFloat(offset : Nat64) : Float = (prim "stableMemoryLoadFloat" : Nat64 -> Float) offset;
 
+/// @deprecated M0199
 func stableMemoryStoreFloat(offset : Nat64, val : Float) : () = (prim "stableMemoryStoreFloat" : (Nat64, Float) -> ())(offset, val);
 
+/// @deprecated M0199
 func stableMemoryLoadBlob(offset : Nat64, size : Nat) : Blob = (prim "stableMemoryLoadBlob" : (Nat64, Nat) -> Blob)(offset, size);
 
+/// @deprecated M0199
 func stableMemoryStoreBlob(offset : Nat64, val : Blob) : () = (prim "stableMemoryStoreBlob" : (Nat64, Blob) -> ())(offset, val);
 
 // Returns a query that computes the current actor's stable variable statistics (for now, the current size, in bytes, of serialized stable variable data).

src/wasm-exts/customModuleEncode.ml

@@ -135,7 +135,7 @@ let encode (em : extended_module) =
       let source_code =
         try
           if filename = "prelude" then Prelude.prelude else
-          if filename = "prim" then Prelude.prim_module ~timers:!Mo_config.Flags.global_timer else
+          if filename = "prim" then Prelude.prim_module  ~timers:!Mo_config.Flags.global_timer else
           (*
           Here we opportunistically see if we can find the source file
           mentioned in the source location, and if we can, include its source

test/bench/ok/region0-mem.tc.ok

@@ -0,0 +1,2 @@
+region0-mem.mo:7.4-7.27: warning [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.

test/bench/ok/stable-mem.tc.ok

@@ -0,0 +1,2 @@
+stable-mem.mo:7.4-7.27: warning [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.

test/fail/ok/illegal-await.tc.ok

@@ -24,14 +24,14 @@ illegal-await.mo:24.11: info, start of scope $@anon-async-24.11 mentioned in err
 illegal-await.mo:26.5: info, end of scope $@anon-async-24.11 mentioned in error at illegal-await.mo:25.7-25.14
 illegal-await.mo:22.10: info, start of scope $@anon-async-22.10 mentioned in error at illegal-await.mo:25.7-25.14
 illegal-await.mo:27.3: info, end of scope $@anon-async-22.10 mentioned in error at illegal-await.mo:25.7-25.14
-illegal-await.mo:35.11-35.12: type error [M0087], ill-scoped await: expected async type from current scope $Rec, found async type from other scope $__18
+illegal-await.mo:35.11-35.12: type error [M0087], ill-scoped await: expected async type from current scope $Rec, found async type from other scope $__15
   scope $Rec is illegal-await.mo:33.44-40.2
-  scope $__18 is illegal-await.mo:33.1-40.2
+  scope $__15 is illegal-await.mo:33.1-40.2
 illegal-await.mo:33.44: info, start of scope $Rec mentioned in error at illegal-await.mo:35.5-35.12
 illegal-await.mo:40.1: info, end of scope $Rec mentioned in error at illegal-await.mo:35.5-35.12
-illegal-await.mo:33.1: info, start of scope $__18 mentioned in error at illegal-await.mo:35.5-35.12
-illegal-await.mo:40.1: info, end of scope $__18 mentioned in error at illegal-await.mo:35.5-35.12
+illegal-await.mo:33.1: info, start of scope $__15 mentioned in error at illegal-await.mo:35.5-35.12
+illegal-await.mo:40.1: info, end of scope $__15 mentioned in error at illegal-await.mo:35.5-35.12
 illegal-await.mo:38.20-38.21: type error [M0096], expression of type
-  async<$__18> ()
+  async<$__15> ()
 cannot produce expected type
   async<$Rec> ()

test/run-drun/experimental-stable-memory--1.mo

@@ -0,0 +1,15 @@
+//MOC-FLAG --experimental-stable-memory -1
+import P "mo:⛔";
+
+import {stableMemoryGrow= _} "mo:⛔";
+actor {
+
+  let _ = P.stableMemorySize;
+}
+
+//SKIP run
+//SKIP run-low
+//SKIP run-ir
+// too slow on ic-ref-run:
+//SKIP comp-ref
+

test/run-drun/experimental-stable-memory-0.mo

@@ -0,0 +1,15 @@
+//MOC-FLAG --experimental-stable-memory 0
+import P "mo:⛔";
+
+import {stableMemoryGrow= _} "mo:⛔";
+actor {
+
+  let _ = P.stableMemorySize;
+}
+
+//SKIP run
+//SKIP run-low
+//SKIP run-ir
+// too slow on ic-ref-run:
+//SKIP comp-ref
+

test/run-drun/experimental-stable-memory-1.mo

@@ -0,0 +1,15 @@
+//MOC-FLAG --experimental-stable-memory 1
+import P "mo:⛔";
+
+import {stableMemoryGrow= _} "mo:⛔";
+actor {
+
+  let _ = P.stableMemorySize;
+}
+
+//SKIP run
+//SKIP run-low
+//SKIP run-ir
+// too slow on ic-ref-run:
+//SKIP comp-ref
+

test/run-drun/ok/experimental-stable-memory--1.tc.ok

@@ -0,0 +1,2 @@
+experimental-stable-memory--1.mo:4.9-4.28: type error [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.

test/run-drun/ok/experimental-stable-memory--1.tc.ret.ok

@@ -0,0 +1 @@
+Return code 1

test/run-drun/ok/experimental-stable-memory-0.drun-run.ok

@@ -0,0 +1,2 @@
+ingress Completed: Reply: 0x4449444c016c01b3c4b1f204680100010a00000000000000000101
+ingress Completed: Reply: 0x4449444c0000

test/run-drun/ok/experimental-stable-memory-0.tc.ok

@@ -0,0 +1,2 @@
+experimental-stable-memory-0.mo:4.9-4.28: warning [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.

test/run-drun/ok/experimental-stable-memory-1.drun-run.ok

@@ -0,0 +1,2 @@
+ingress Completed: Reply: 0x4449444c016c01b3c4b1f204680100010a00000000000000000101
+ingress Completed: Reply: 0x4449444c0000

test/run-drun/ok/gc-random-test-force-gc.tc.ok

@@ -0,0 +1,2 @@
+gc-random-test/types.mo:301.20-301.41: warning [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.

test/run-drun/ok/gc-random-test-no-force-gc.tc.ok

@@ -0,0 +1,2 @@
+gc-random-test/types.mo:301.20-301.41: warning [M0199], this code is (or uses) the deprecated library `ExperimentalStableMemory`.
+Please use the `Region` library instead: https://internetcomputer.org/docs/current/motoko/main/stable-memory/stable-regions/#the-region-library or compile with flag `--experimental-stable-memory 1` to suppress this message.