Enhanced Orthogonal Persistence (Complete Integration) (#4488)

* Prepare two compilation targets * Combined RTS Makefile * Port classical compiler backend to combined solution * Adjust nix config file * Start combined RTS * Reduce classical compiler backend changes * Continue combined RTS * Make RTS compilable for enhanced orthogonal persistence * Make RTS tests runnable again for enhanced orthogonal persistence * Adjust compiler backend of enhanced orthogonal persistence * Unify Tom's math library binding * Make classical non-incremental RTS compile again * Make classical incremental GC version compilable again * Make all RTS versions compile again * Adjust memory sanity check for combined RTS modes * Prepare RTS tests for combined modes * Continue RTS test merge * Continue RTS tests combined modes * Continue RTS tests support for combined modes * Adjust LEB128 encoding for combined mode * Adjust RTS test for classical incremental GC * Adjust RTS GC tests * Different heap layouts in RTS tests * Continue RTS GC test multi-mode support * Make all RTS run again * Adjust linker to support combined modes * Adjust libc import in RTS for combined mode * Adjust RTS test dependencies * Bugfix in Makefile * Adjust compiler backend import for combined mode * Adjust RTS import for combined mode * Adjust region management to combined modes * Adjust classical compiler backend to fit combined modes * Reorder object tags to match combined RTS * Adjust test * Adjust linker for multi memory during Wasi mode with regions * Adjust tests * Adjust bigint LEB encoding for combined modes * Adjust bigint LEB128 encoding for combined modes * Adjust test * Adjust tests * Adjust test * Code refactoring: SLEB128 for BigInt * Adjust tests * Adjust test * Reformat * Adjust tests * Adjust benchmark results * Adjust RTS for unit tests * Reintroduce compiler flags in classical mode * Support classical incremental GC * Add missing export for classical incremental GC * Adjust tests * Adjust test * Adjust test * Adjust test * Adjust test * Adjust test * Adjust test * Pass `keep_main_memory` upgrade option only for enhanced orthogonal persistence * Adjust test * Update nix hash * Adjust Motoko base dependency * Adjust tests * Extend documentation * Adjust test * Update documentation * Update documentation * Manual merge conflict resolution * Manual merge refinement * Manual merge conflict resolution * Manual merge conflict resolution * Refactor migration test from classical to new persistence * Adjust migration test * Manual merge conflict resolution * Manual merge conflict resolution * Adjust compiler reference documentation * Test CI build * Test CI build * Adjust performance comparison in CI build * Manual merge conflict resolution * Add test for migration paths * Adjust test for integrated PR * Adjust test case * Manual merge conflict resolution * Manual merge conflict resolution * Manual merge conflict resolution * Manual merge conflict resolution * Code refactoring * Fix typo in comment Co-authored-by: Claudio Russo <claudio@dfinity.org> * Manual merge conflict resolution * Add static assertions, code formatting * Manual merge conflict resolution * Add test case * Refine comment Co-authored-by: Claudio Russo <claudio@dfinity.org> * Manual merge conflict resolution * Manual merge conflict resolution * Code refactoring * Manual merge conflict resolution * Adjust test run script messages * Manual merge conflict resolution * Manual merge conflict resolution * Manual merge conflict resolution * Manual merge conflict resolution * Merge Preparation: Dynamic Memory Capacity for Integrated EOP (#4586) * Tune for unknown memory capacity in 64-bit * Adjust benchmark results * Fix debug assertion, code refactoring * Manual merge conflict resolution * Manual merge conflict resolution * Code refactoring: Improve comments * Reformat * Fix debug assertion * Re-enable memory reserve for upgrade and queries See PR #4158 * Manual merge conflict resolution * Manual merge conflict resolution * Update benchmark results * Manual merge conflict resolution * Manual merge conflict resolution * Merge Preparation: Latest IC with Integrated EOP (#4638) * Adjust to new system API * Port to latest IC 64-bit system API * Update to new IC with Wasm64 * Updating nix hashes * Update IC dependency (Wasm64 enabled) * Update expected test results * Fix migration test * Use latest `drun` * Adjust expected test results * Updating nix hashes * Update expected test results * Fix `drun` nix build for Linux * Disable DTS in `drun`, refactor `drun` patches * Update expected test results for new `drun` * Limiting amount of stable memory accessed per graph copy increment * Reformat * Manual merge conflict resolution * Manual merge conflict resolution * Adjust expected test result --------- Co-authored-by: Nix hash updater <41898282+github-actions[bot]@users.noreply.github.com> * Manual merge conflict resolution * Documentation Update for Enhanced Orthogonal Persistence (#4670) --------- Co-authored-by: Claudio Russo <claudio@dfinity.org> Co-authored-by: Nix hash updater <41898282+github-actions[bot]@users.noreply.github.com>
dfinity · Aug 26, 2024 · 320fe4b · 320fe4b
1 parent ce94118
commit 320fe4b
Show file tree

Hide file tree

Showing 328 changed files with 22,942 additions and 1,775 deletions.
diff --git a/bin/wrapper.sh b/bin/wrapper.sh
@@ -18,7 +18,7 @@ declare -A envs # list of expected environment variables with paths to products
 # Define build products here
 real[moc]=src/moc
 hint[moc]="make -C $rel_root/src moc"
-envs[moc]="MOC_RELEASE_RTS MOC_DEBUG_RTS"
+envs[moc]="MOC_NON_INCREMENTAL_RELEASE_RTS MOC_NON_INCREMENTAL_DEBUG_RTS MOC_INCREMENTAL_RELEASE_RTS MOC_INCREMENTAL_DEBUG_RTS MOC_EOP_RELEASE_RTS MOC_EOP_DEBUG_RTS"
 real[mo-ld]=src/mo-ld
 hint[mo-ld]="make -C $rel_root/src mo-ld"
 real[mo-doc]=src/mo-doc
@@ -32,8 +32,12 @@ hint[candid-tests]="make -C $rel_root/src candid-tests"
 
 rts_hint="make -C $rel_root/rts"
 
-real[MOC_RELEASE_RTS]=rts/mo-rts.wasm
-real[MOC_DEBUG_RTS]=rts/mo-rts-debug.wasm
+real[MOC_NON_INCREMENTAL_RELEASE_RTS]=rts/mo-rts-non-incremental.wasm
+real[MOC_NON_INCREMENTAL_DEBUG_RTS]=rts/mo-rts-non-incremental-debug.wasm
+real[MOC_INCREMENTAL_RELEASE_RTS]=rts/mo-rts-incremental.wasm
+real[MOC_INCREMENTAL_DEBUG_RTS]=rts/mo-rts-incremental-debug.wasm
+real[MOC_EOP_RELEASE_RTS]=rts/mo-rts-eop.wasm
+real[MOC_EOP_DEBUG_RTS]=rts/mo-rts-eop-debug.wasm
 
 for var in ${envs[moc]}; do
   hint[$var]=$rts_hint

diff --git a/default.nix b/default.nix
@@ -198,7 +198,7 @@ rec {
         name = "motoko-rts-deps";
         src = subpath ./rts;
         sourceRoot = "rts/motoko-rts-tests";
-        sha256 = "sha256-YEw2AFi15JVKP7owsPDoBT3qSegOO90FRn2qoUBAICw=";
+        sha256 = "sha256-prLZVOWV3BFb8/nKHyqZw8neJyBu1gs5d0D56DsDV2o=";
         copyLockfile = true;
       };
 
@@ -261,8 +261,12 @@ rec {
 
       installPhase = ''
         mkdir -p $out/rts
-        cp mo-rts.wasm $out/rts
-        cp mo-rts-debug.wasm $out/rts
+        cp mo-rts-non-incremental.wasm $out/rts
+        cp mo-rts-non-incremental-debug.wasm $out/rts
+        cp mo-rts-incremental.wasm $out/rts
+        cp mo-rts-incremental-debug.wasm $out/rts
+        cp mo-rts-eop.wasm $out/rts
+        cp mo-rts-eop-debug.wasm $out/rts
       '';
 
       # This needs to be self-contained. Remove mention of nix path in debug
@@ -272,7 +276,17 @@ rec {
           -t ${nixpkgs.rustc-nightly} \
           -t ${rtsDeps} \
           -t ${rustStdDeps} \
-          $out/rts/mo-rts.wasm $out/rts/mo-rts-debug.wasm
+          $out/rts/mo-rts-non-incremental.wasm $out/rts/mo-rts-non-incremental-debug.wasm
+        remove-references-to \
+          -t ${nixpkgs.rustc-nightly} \
+          -t ${rtsDeps} \
+          -t ${rustStdDeps} \
+          $out/rts/mo-rts-incremental.wasm $out/rts/mo-rts-incremental-debug.wasm
+        remove-references-to \
+          -t ${nixpkgs.rustc-nightly} \
+          -t ${rtsDeps} \
+          -t ${rustStdDeps} \
+          $out/rts/mo-rts-eop.wasm $out/rts/mo-rts-eop-debug.wasm
       '';
 
       allowedRequisites = [];
@@ -363,6 +377,31 @@ rec {
           EXTRA_MOC_ARGS = "--sanity-checks";
       };
 
+    snty_compacting_gc_subdir = dir: deps:
+      (test_subdir dir deps).overrideAttrs {
+          EXTRA_MOC_ARGS = "--sanity-checks --compacting-gc";
+      };
+
+    snty_generational_gc_subdir = dir: deps:
+      (test_subdir dir deps).overrideAttrs {
+          EXTRA_MOC_ARGS = "--sanity-checks --generational-gc";
+      };
+
+    snty_incremental_gc_subdir = dir: deps:
+      (test_subdir dir deps).overrideAttrs {
+          EXTRA_MOC_ARGS = "--sanity-checks --incremental-gc";
+      };
+
+    enhanced_orthogonal_persistence_subdir = dir: deps:
+      (test_subdir dir deps).overrideAttrs {
+          EXTRA_MOC_ARGS = "--enhanced-orthogonal-persistence";
+      };
+
+    snty_enhanced_orthogonal_persistence_subdir = dir: deps:
+      (test_subdir dir deps).overrideAttrs {
+          EXTRA_MOC_ARGS = "--sanity-checks --enhanced-orthogonal-persistence";
+      };
+
     perf_subdir = dir: deps:
       (test_subdir dir deps).overrideAttrs (args: {
         checkPhase = ''
@@ -474,15 +513,27 @@ rec {
   in fix_names ({
       run        = test_subdir "run"        [ moc ] ;
       run-dbg    = snty_subdir "run"        [ moc ] ;
+      run-eop-release = enhanced_orthogonal_persistence_subdir "run" [ moc ];
+      run-eop-debug = snty_enhanced_orthogonal_persistence_subdir "run" [ moc ];
       # ic-ref-run = test_subdir "run-drun"   [ moc ic-ref-run ];
       drun       = test_subdir "run-drun"   [ moc nixpkgs.drun ];
       drun-dbg   = snty_subdir "run-drun"   [ moc nixpkgs.drun ];
+      drun-compacting-gc = snty_compacting_gc_subdir "run-drun" [ moc nixpkgs.drun ] ;
+      drun-generational-gc = snty_generational_gc_subdir "run-drun" [ moc nixpkgs.drun ] ;
+      drun-incremental-gc = snty_incremental_gc_subdir "run-drun" [ moc nixpkgs.drun ] ;
+      drun-eop-release = enhanced_orthogonal_persistence_subdir "run-drun" [ moc nixpkgs.drun ] ;
+      drun-eop-debug = snty_enhanced_orthogonal_persistence_subdir "run-drun" [ moc nixpkgs.drun ] ;
       fail       = test_subdir "fail"       [ moc ];
+      fail-eop   = enhanced_orthogonal_persistence_subdir "fail"       [ moc ];
       repl       = test_subdir "repl"       [ moc ];
+      repl-eop   = enhanced_orthogonal_persistence_subdir "repl"       [ moc ];
       ld         = test_subdir "ld"         ([ mo-ld ] ++ ldTestDeps);
+      ld-eop     = enhanced_orthogonal_persistence_subdir "ld" ([ mo-ld ] ++ ldTestDeps);
       idl        = test_subdir "idl"        [ didc ];
       mo-idl     = test_subdir "mo-idl"     [ moc didc ];
+      mo-idl-eop = enhanced_orthogonal_persistence_subdir "mo-idl" [ moc didc ];
       trap       = test_subdir "trap"       [ moc ];
+      trap-eop   = enhanced_orthogonal_persistence_subdir "trap" [ moc ];
       run-deser  = test_subdir "run-deser"  [ deser ];
       perf       = perf_subdir "perf"       [ moc nixpkgs.drun ];
       bench      = perf_subdir "bench"      [ moc nixpkgs.drun ic-wasm ];

diff --git a/design/Custom-Sections.md b/design/Custom-Sections.md
@@ -34,3 +34,7 @@ let hash : string -> int32 = fun s ->
       (Lib.String.explode s)
   )
 ```
+
+Motoko generates an additional `"enhanced-orthogonal-persistence"` private custom section to
+mark Motoko Wasm binaries that rely on IC's support to retain the main Wasm memory on an upgrade, 
+cf. [Orthogonal Persistence](OrthogonalPersistence.md).
diff --git a/design/DFX-Interface.md b/design/DFX-Interface.md
@@ -118,6 +118,10 @@ used only in very specific cases.
 The above metadata is stored in the Wasm module, and is only accessible by the controllers of the canister, unless the
 metadata name is specified in the `--public-metadata` flag.
 
+Moreover, the compiler generates a special marker custom section `"enhanced-orthogonal-persistence"` if the new orthogonal 
+persistence support is enabled, see [Orthogonal Persistence](OrthogonalPersistence.md). This section is always private and
+always emited independent of the compiler flags `--public-metadata` or `--public-metadata`.
+
 Checking stable type compatibility
 ----------------------------------
 
@@ -130,6 +134,10 @@ a type safe way without unintentional data loss.
 
 If the check succeeds, nothing will be printed. 
 If the check fails, the error message will be printed in stderr and the command returns with exit code 1.
+The check can also emit warning messages, e.g. if stable variables are dropped.
+
+With [enhanced orthogonal persistence](OrthogonalPersistence.md), the stable compatibility is also integrated in the runtime
+system, to atomically guarantee memory compatibility during an upgrade.
 
 Invoking the IDE
 ----------------

diff --git a/design/GraphCopyStabilization.md b/design/GraphCopyStabilization.md
@@ -1,15 +1,12 @@
 # Graph-Copy-Based Stabilization
 
-This is part of the enhanced orthogonal persistence support, see `OrthogonalPersistence.md`.
-It allows future potentially radical changes of the persistent memory layout, such as introducing a new GC, rearranging persistent metadata, or specializing arrays for small element types etc. 
+This is part of the enhanced orthogonal persistence support, see [Orthogonal Persistence](OrthogonalPersistence.md).
 
 ## Purpose
-
-This allows potentially radical changes of the persistent main memory layout, e.g. introducing a new GC or rearranging persistent metadata. 
+This allows future potentially radical changes of the persistent memory layout, such as introducing a new GC, rearranging persistent metadata, or specializing arrays for small element types etc. 
 This also relies on precise value tagging to allow more advanced changes that require value metadata, e.g. specializing arrays for small value element types or even downgrading to 32-bit heap layouts (provided that the amount of live data fits into a 32-bit memory).
 
 ## Design
-
 Graph copy of sub-graph of stable objects from main memory to stable memory and vice versa on upgrades.
 
 ## Properties
@@ -22,9 +19,7 @@ Graph copy of sub-graph of stable objects from main memory to stable memory and
 ## Memory Compatibility Check
 Apply a memory compatibility check analogous to the enhanced orthogonal persistence, since the upgrade compatibility of the graph copy is not identical to the Candid subtype relation.
 
-
 ## Incremental Upgrade
-
 Supporting arbitrarily large upgrades beyond the instruction limit:
 * Splitting the stabilization/destabilization in multiple asynchronous messages.
 * Limiting the stabilization work units to fit the update or upgrade messages.
@@ -35,7 +30,6 @@ Supporting arbitrarily large upgrades beyond the instruction limit:
 **Note**: Graph copying needs to be explicitly initiated as the usual upgrade engages enhanced orthogonal persistence, simply retaining main memory with compatibility check.
 
 ### Usage
-
 When upgrading to a Motoko version that is not compatible with the current enhanced orthogonal persistence:
 
 1. Initiate the explicit stabilization before the upgrade:
@@ -76,15 +70,23 @@ dfx canister call CANISTER_ID __motoko_destabilize_after_upgrade "()"
 * The GC is restarted.
 
 ### Remarks
-
-* Steps 2 (explicit destabilization) may not be needed if the corresponding operation fits into the upgrade message.
+* When receiving the `dfx` error "The request timed out." during explicit stabilization, upgrade, or destabilization, one can simply repeat the call until it completes.
+* Steps 3 (explicit destabilization) may not be needed if the corresponding operation fits into the upgrade message.
 * Stabilization and destabilization steps are limited to the increment limits:
 
     Operation | Message Type | IC Instruction Limit | **Increment Limit**
     ----------|--------------|----------------------|--------------------
     **Explicit (de)stabilization step** | Update | 20e9 | **16e9**
     **Actual upgrade** | Upgrade | 200e9 | **160e9**
 
+* The graph copy steps also limit the amount of processed stable data (read or write), in order not to exceed the 
+IC's stable memory access limits.
+
+    Operation | Message Type | IC Stable Access Limit | **Increment Limit**
+    ----------|--------------|----------------------|--------------------
+    **Explicit (de)stabilization step** | Update | 2 GB | **1 GB**
+    **Actual upgrade** | Upgrade | 8 GB | **6 GB**
+
 ## Graph-Copy Algorithm
 Applying Cheney’s algorithm [1, 2] for both serialization and deserialization:
 
@@ -117,6 +119,7 @@ The format is also versioned to allow future refinements of the graph copy algor
 * Incremental GC: Serialization needs to consider Brooks forwarding pointers (not to be confused with the Cheney's forwarding information), while deserialization can deal with partitioned heap that can have internal fragmentation (free space at partition ends).
 * The partitioned heap prevents linear scanning of the heap, especially in the presence of large objects that can be placed at a higher partition than subsequently allocated normal-sized objects. For this reason, a scan stack is allocated in the main memory, remembering the deserialized objects that still need to be scanned. With this, the deserialization does not need to make any assumptions of the heap structure (e.g. monotonically increasing allocations, free space markers, empty heap on deserialization start etc.).
 * If actor fields are promoted to the `Any` type in a new program version, their content is released in that variable to allow memory reclamation.
+* Both stabilization and destabilization read and write data linearly, which is beneficial for guarding a work set limit (number of accessed pages) per IC message. Destabilization is also linear because it deserializes objects in the same order back as they have been serialized.
 
 ## Open Aspects
 * Unused fields in stable records that are no longer declared in a new program versions should be removed. This could be done during garbage collection, when objects are moved/evacuated. This scenario equally applies to enhanced orthogonal persistence.

diff --git a/design/Implementation.md b/design/Implementation.md
@@ -9,25 +9,29 @@
 
 ## Heap
 
-* Uniform representation with 32 bit word size.
+* Uniform representation with a defined word size. 
+For [enhanced orthogonal persistence](OrthogonalPersistence.md), 64-bit words are used, while for classical persistence, the word size is 32-bit.
 
-* Use pointer tagging in LSB;.
-  - 0 for pointers, 1 for scalars.
-  - Scalars are real value shifted left by 1, lowest bit set.
-
-* Q: Allocation and GC strategies?
+* Use pointer tagging in the LSB:
+  - 1 for pointers, 0 for scalars.
+  - Scalars are real value shifted left by 1, lowest bit clear.
+  For [enhanced orthogonal persistence](OrthogonalPersistence.md), the types of scalars are additionally tagged.
+
+* Garbage collected.
 
 
 ## Primitive types
 
 * Nat and Int compile to heap-allocated big nums; unboxed for small numbers `<= 31` bit.
 
-* Nat8/16 compile to unboxed scalars; Nat32/64 are boxed.
+* Nat8/16 compile to unboxed scalars; 
+  On a 32-bit heap, Nat32/64 are boxed. 
+  On a 64-bit heap, only Nat64 is boxed, while Nat32 remains unboxed.
   - May unbox locally.
 
 * Characters are scalars (unicode code points).
 
-* Text is heap-allocated.
+* Text is heap-allocated. Using ropes for concatenations.
 
 
 ## Tuples
@@ -103,6 +107,11 @@ TODO
 
 TODO
 
+## Persistence
+
+Different * [persistence modes](OrthogonalPersistence.md):
+* [Enhanced orthogonal persistence](OrthogonalPersistence.md).
+* [Classical persistence](OldStableMemory.md).
 
 # Hypervisor Extensions needed