Add integration testing tooling setup for JIT

[thecrypticace]

Tools:

• Hugo
• Laravel Mix v6
• Next
• Nuxt
• Parcel 2
• PostCSS CLI
• Snowpack
• Vite
• webpack 5 + postcss
• webpack 5 + sass/less/etc…
• Webpacker

Actions:

• Initial Build
• Add utility to html files
• Add utility to vue files
• Add utility to react/jsx files
• Add @​apply
• Update @​apply
• Change config

There are a few things I have to figure out:

• Proper process cleanup — if a process is spawned that is not an apparent child process Jest does not exit but killing the process that spawned it doesn't clean up either. This affects the Laravel Mix which is otherwise also "working".
• Handling known / expected tooling failures

[thecrypticace]

I need to adjust the actions scripts to install npm dependencies for the tooling directories.

@RobinMalfait Do you know of / can think of a good way to handle these given that we're caching node deps here?

[RobinMalfait]

I think there are a few things you can do. We could create individual workflows for each tool. We could also setup steps for each tool.

If you look at the current GitHub actions, then you will see this:

- name: Use cached node_modules
  id: cache  # This is the ID used below
  uses: actions/cache@v2.1.1
  with:
    path: node_modules
    key: nodeModules-${{ hashFiles('**/package-lock.json') }}-${{ matrix.node-version }}
    restore-keys: |
      nodeModules-
- name: Install dependencies
  if: steps.cache.outputs.cache-hit != 'true'  # This is referring to steps.ID-OF-STEP.outputs.cache-hit
  run: npm install
  env:
    CI: true

So we could create a step for each tool, one way to do it is to define a matrix for each tool. We already do that for the node-version:

strategy:
  matrix:
    node-version: [12.x, 14.x]

Then we use that node-version inside the steps below. For example we can define a matrix for each node-version (currently there) and for each tool. Example: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-operating-system

Note, this could result in a lot of combinations 😅

If we go with this approach, then we should also be able to run them in parallel (I think that's the default behaviour).

I think the only thing left to do is to also encode the tool in the key field when caching:

- name: Use cached node_modules
  id: cache-${{ matrix.tool }}  # This is the ID used below
  uses: actions/cache@v2.1.1
  with:
    path: node_modules
    key: nodeModules-${{ hashFiles('**/package-lock.json') }}-${{ matrix.node-version }}-${{ matrix.tool }}
    restore-keys: |
      nodeModules-
- name: Install dependencies
  if: steps.cache-${{ matrix-tool }}.outputs.cache-hit != 'true'  # This is referring to steps.ID-OF-STEP.outputs.cache-hit
  run: npm install
  env:
    CI: true

I haven't tested it, and we might need to change this approach if we have a lot of tools to test, but I think this can work. What do you think?

[thecrypticace]

Yeah a separate GitHub action would work. We wouldn't necessarily need to to a matrix build (but could). It could end up being really easy to inch close to the 2,000 minute / month limit running all these integration tests and I want to be mindful of that.

[thecrypticace]

Seems like CI is having problems with the main tests. They're taking a really really long time to even start up. Perhaps related to test coverage checking?

[adamwathan]

Superseded by #4354 — thanks though @thecrypticace you are the best man ❤️