[code-infra] Refactor Netlify cache-docs plugin setup
#14105
Conversation
|
Deploy preview: https://deploy-preview-14105--material-ui-x.netlify.app/ |
|
|
||
| [build.environment] | ||
| NODE_VERSION = "18" | ||
| NODE_OPTIONS = "--max_old_space_size=4096" | ||
| PNPM_FLAGS = "--shamefully-hoist" | ||
|
|
||
| [[plugins]] | ||
| package = "./node_modules/@mui/monorepo/packages/netlify-plugin-cache-docs" |
There was a problem hiding this comment.
🤔 Out of scope, but we should try the same trick as we do with eslint-plugin-material-ui instead of accessing node_modules directly
There was a problem hiding this comment.
I've tried a minimal setup with the main requirements—manifest file and re-exporting the @mui/monorepo package.
WDYT? 🤔
There was a problem hiding this comment.
Have you tested if this solves the Module not found: Can't resolve '@mui/joy/InitColorSchemeScript' issue?
There was a problem hiding this comment.
I was under the impression that there is nothing to test in that regard.
The issue is the same that we had before bumping @mui/material to unblock the bump of @mui/monorepo and is repeated when a stale cache is picked up. 🙈
Do you have reasons to believe that the culprit of the failure are different? 🤔
There was a problem hiding this comment.
Hmm, I forgot about that one.
Do you have reasons to believe that the culprit of the failure are different? 🤔
I'm not sure it's only about caching, because we do run pnpm install after the cache is restored.
So it shouldn't matter if the cache is stale, it should be invalidated and dependencies should be properly resolved.
Right?
I was looking for the problem on our side, and this is why I opened #14104. But perhaps this was just a coincidence, I'm confused now 😅
There was a problem hiding this comment.
I'm not sure it's only about caching, because we do run
pnpm installafter the cache is restored.
So it shouldn't matter if the cache is stale, it should be invalidated and dependencies should be properly resolved.
Right?
I was looking for the problem on our side, and this is why I opened #14104. But perhaps this was just a coincidence, I'm confused now 😅
If only mui-x was having problems, I would have also investigated possible problems on our end.
But the problem is that both material-ui and mui-x started having the same problems at the same time. 🤔
There was a problem hiding this comment.
I still don't understand how stale cache can survive pnpm install though 🤔
There was a problem hiding this comment.
Maybe we should try downgrading pnpm?
There was a problem hiding this comment.
I still don't understand how stale cache can survive pnpm install though 🤔
Netlify caches everything.
Maybe the installation problem gets fixed after running pnpm install, but the build files (.next directory) have also been saved and could make the build fail depending on how Netlify manages this case... 🙈
WDYT, does that sound feasible?
We could consider no longer caching the .next folder, but that would slow down many deploys significantly. 🙈 🤷
There was a problem hiding this comment.
I've tested the build both with docs/.next caching and without and it seems like the difference in build time is somewhat negligible. 🙈
1st build is without cache, the second is with it...
WDYT @mui/code-infra, does it make sense to additionally cache docs/.next? 🤔
There was a problem hiding this comment.
I tried the same last week in my local setup and the difference in next build time seemed to be about 1 to 2 ratio. (time pnpm docs:build with and without .next folder present)
The build step (next build) in both your builds takes about 4-5 minutes. I looked at a random other X netlify build and the build step is about 2-3 minutes. Are you sure the one you say is using cache is actually using cache?
(build.command completed in 4m 24.6s)and(build.command completed in 4m 56.8s)for your two builds(build.command completed in 2m 44.8s)for the build of https://app.netlify.com/sites/material-ui-x/deploys/66b23f0206998000085c0eff
(Also, we can probably just cache .next/cache instead of the whole .next folder)
Two more options I'd consider trying:
- downgrade pnpm 1 or 2 minor versions (or any version we were on before the trouble started)
- contact netlify support
edit: best to look at actual build times in the logs. There will be less variation. I think the Netlify overall build time includes queueing as well
There was a problem hiding this comment.
We could consider no longer caching the .next folder
I've tried that, you can find it in the "Disproved theories" section in #14104
LukasTy
changed the title
base folderdocs-cache plugin management
LukasTy
changed the title
docs-cache plugin managementcache-docs plugin management
LukasTy
changed the title
cache-docs plugin managementcache-docs plugin setup
| @@ -1,5 +1,3 @@ | |||
| const fetch = require('node-fetch'); | |||
There was a problem hiding this comment.
ok, this wasn't even in package.json? 😄
There was a problem hiding this comment.
Nope. 🙈
We relied on a transitive dependency. 😆
We are rocking a monorepo and we deploy our documentation Netlify, which is ondocsfolder to netlify.Based on their doc and the following points from our setup might not be the most correct and it could be resulting in flaky cache behavior. 🤔Do you see any problems with it?Is it worth testing it?
Given this doc my initial assumption was incorrect.
Why I tried looking in this?
A recent PR (#14092) failed Netlify deployment.
The final result of this PR is just some refactoring as the initial agenda seemed incorrect given our current setup.
netlify-plugin-cache-docsplugin management by re-exporting it from the internal package and target it, instead of@mui/monorepoin thenetlify.toml.fetchinstead ofnode-fetchindeploy-succeededNetlify function.