Feature: engine to convert ipynb to markdown (via jupytext)

[zkamvar]

Summary

This comes from lessons learned from https://github.com/datacarpentry/astronomy-python/, where Allen Downey has prepared a script for converting from iPython notebooks to markdown.

Since we are using markdown as an intermediate to HTML it should be a matter of dropping in an engine that renders and converts a single jupyter notebook to markdown. The script can be called from the {processx} package.

Tasks

• confirm a python script can be run consistently from R via processx
• create a python engine for conversion
• create tests for python conversion
• add python engine to code, acting on *.myst notebook files
• add optional config for users to insert their own engine

Edit: I'm changing this from ipynb based to myst notebook/jupytext based operations because ipynb format is not git-friendly even if github will render them.

[fmichonneau]

keep this for later. Let's first focus on handling markdown files robustly.

[zkamvar]

This is related to #61

[sabaferdous12]

@zkamvar , Just wanted to mention that we have done implementation of markdown to ipynb and vice versa is also in things to implement soon. After a bit of more testing and fixing little issues on GitHub Actions, will put a merge request :)

[zkamvar]

OMG THANK YOU! I've seen the work and I am impressed with the solution you found!

[sabaferdous12]

Will keep you posted on this :)

[milanmlft]

To convert ipynb to RMarkdown, rmarkdown::convert_ipynb() might be useful. This should retain the evaluated code chunks.

[zkamvar]

Thank you @milanmlft, I think that's a reasonable solution.

I had not considered the fact that ipynb notebooks would store the results. This is not the same as the R Markdown solution, which dynamically renders the results, but it is better than the copy/paste methods lesson authors currently have.

There are two challenges that remain:

1. we would need to write a similar parser like {pegboard} for ipynb to validate and extract markdown from the notebooks or disable it for them
2. notebooks do not play well with git diffs (though the technology may exist and I'm likely not aware of it)

[rbavery]

The main open source option for nb diffs: https://nbdime.readthedocs.io/en/latest/

[milanmlft]

Hi all,

I've been experimenting with setting up an R wrapper for jupytext and have come up with this: milanmlft/jupytextR. In theory, this should allow converting between any of the jupytext-supported file formats from within R.

It's still in an experimental state, but would love to get your feedback on this and was wondering whether this might be useful for sandpaper?

[zkamvar]

It's still in an experimental state, but would love to get your feedback on this and was wondering whether this might be useful for sandpaper?

I apologise for the delay in my response. I realize that I never updated this particular issue: I believe that the myst format (and thus a jupytext engine) is not in scope for this project because there are significant departures from the pandoc markdown syntax that would make it difficult to support. The motivation for suggesting Jupytext came as a suggestion from one of the alpha testers before Quarto was a thing, but I had not dug deep enough into it when I updated this issue.

[milanmlft]

Thanks for the feedback! Didn't realize the differences between Markdown format. We're currently using jupytext in L2D to convert the sandpaper Rmd files to jupyter notebooks, as supplementary material for the students. That seems to be working fine, but of course it's a very specific example.