Define a pipeline of artifacts which target different users of Noir programs

[TomAFrench]

Problem

Noir/Nargo needs to serve multiple artifacts to different consumers while ensuring that all the components of these are in sync (e.g. proving key matches the circuit being proven). The current way we do this is:

circuit.json (acir + ABI)
circuit.json.checksum (consistency check)
circuit.pk (proving key)
circuit.vk (verification key)

This is kinda ok but not amazing. The motivation behind these files being broken out is that the proving keys can be quite large and so we don't want to pass this to, for instance, a verifier server as they don't need it. This then means that we need a checksum file to ensure that the keys are in sync with the circuit.

We also need to consider Noir contracts. For instance a contract with functions foo, bar and baz will end up with the following artifacts.

contract.json (acir + ABI + vk for each contract function)
circuit.foo.json.checksum
circuit.bar.json.checksum
circuit.baz.json.checksum
circuit.foo.pk
circuit.bar.pk
circuit.baz.pk
circuit.foo.vk
circuit.bar.vk
circuit.baz.vk

This is Not Great^TM:

• We've got a separate checksum for every contract function.
• For a contract with many functions we need to scoop up a whole directory full of files to pass them off to a prover.
• If a contract function is removed then its proving and verification keys will currently hang around until explicitly cleared.

Currently contract.json's format is dictated by Aztec 3 and doesn't include the pks for similar reasons that pks and vks are omitted from circuit.json for Noir programs.

This system is generally clunky to work with and also has an abstraction leak from Aztec 3. As Noir is neutral to Aztec vs other consumers we can't directly output Aztec 3 artifacts without constraining other consumers.

Proposed solution

Rather than directly outputting the artifacts which are decomposed to the point where different users need to collect all of the files which are relevant to themselves, we should create a defined artifact pipeline which starts of general and then allows users to pull out subsets which are relevant to them. This would be done in code rather than manually pulling out fields from the json files.

For instance the flow for a Noir program would looks something along the lines of

flowchart TD
    A[Noir Source] -->|compile| B(Backend-Agnostic Artifact \n ACIR + ABI)
    B -->|preprocess| C(Backend-specific Artifact \n optimised ACIR + ABI + PK + VK)
 
    C -->|extract| D(Prover Artifact \n ACIR + ABI + PK)
    C -->|extract| E(Verifier Artifact \n ACIR hash + ABI + VK)

Loading

This has the benefit for knowing that any artifact is internally consistent (unless someone manually mucks with it) so there's no need for checksums. Something to note is that Nargo currently produces optimised bytecode by default (#658) so we must skip the first artifact type for now until that's fixed.

Similarly for a Noir contract we would have a flow similar to

flowchart TD
    A[Noir Source] -->|compile| B(Backend-Agnostic Artifact \n ACIR + ABI)
    B -->|preprocess| C(Backend-specific Artifact \n optimised ACIR + ABI + PK + VK)
 
    C -->|extract| D(Aztec Contract Artifact \n ACIR + ABI + VK)
    C -->|extract| E(NotAztec Contract Artifact \n ACIR + ABI + VK + extras)

Loading

(each artifact would contain many ACIRs, ABIs, keys but let's ignore that for now)

This includes the contract artifact for the imaginatively named "NotAztec" project which doesn't conform to the same contract artifact interface as Aztec 3 but does use Noir/ACIR for its contracts.

Alternatives considered

We could stay similarly to how we are now and emit the individual components of these build artifacts and have it be the responsibility of users to combine them in useful ways. I'd argue that the fact that we've changed the contract output to match that expected by Aztec 3 is a sign that this is not workable.

Additional context

No response

Submission Checklist

• Once I hit submit, I will assign this issue to the Project Board with the appropriate tags.

[guipublic]

I don't think backend-agnostic artefacts are desirable, they are more difficult to handle for noir, which will lead to less efficient code, and at the same time they are not that useful for users, as it is only helpful in early phase when a user wants to investigate the backend that is suitable for its use case, or for users that would like to benchmark backends.

[TomAFrench]

Putting aside generating generic artifacts, this work is desirable as it allows us to decouple the compiler frontend/evaluator from backends which leads to significant simplification of the frontend.

Currently we need to drill down into the depths of the compiler to feed in a Box<dyn Fn(&Opcode) -> bool> in various locations (see #1349). This is complicated by the fact that we can't create multiple instances of this boxed closure easily due to lifetimes. We otherwise need to make NodeInterner generic which then becomes viral across much of the compiler.

If we postponed all backend-specific handling of opcodes to occur in acvm::compile then the compiler would be simpler. Generic artifacts then fall out as a side-benefit.

they are more difficult to handle for noir, which will lead to less efficient code

Can you give an example of how this is more difficult? I'm not clear on how this would cause problems (afaik replacing our noir fallback functions in ACVM is a little bit of a pain but we have multiple ways to address this).

[guipublic]

I don't understand why having a function which just tells which opcodes are supported leads to significant complexity and generate lifetime issues. If it is true, then there is something we do wrong and removing the function is not the solution.

If you want to generate backend-agnostic ACIR from noir, then you should first remove any field instantiation from noir.
Please give it a try, this should show you why it is difficult to do (or this would prove me wrong).

[kevaundray]

If a user calls noir the compiler to compile noir source code, then they should not need to have a backend. Noir would then output an ACIR which is non-optimal for a particular backend and then nargo can then choose to make it backend specific, or perhaps the user can take this ACIR format and do whatever they want with it.

field-agnostic and backend-agnostic are not the same thing here since multiple backends can use the same field. The idea above is more about making sure the Noir compiler has no reliance on a backend, it can have a reliance on a particular Field and if a backend does not support said Field then it would say it when we try to make the ACIR backend-specific.

[guipublic]

, or perhaps the user can take this ACIR format and do whatever they want with it.

What use case do you have in mind?

making sure the Noir compiler has no reliance on a backend

This sentence is wrong. While I agree field-agnostic and backend-agnostic are not the same thing, they are strongly related. This is why you do not have the Noir compiler having no reliance on the backend because it has reliance on the field.