What's the best way to do literate programming in Python on Windows?

Question

I've been playing with various ways of doing literate programming in Python. I like noweb, but I have two main problems with it: first, it is hard to build on Windows, where I spend about half my development time; and second, it requires me to indent each chunk of code as it will be in the final program --- which I don't necessarily know when I write it. I don't want to use Leo, because I'm very attached to Emacs.

Is there a good literate programming tool that:

1. Runs on Windows
2. Allows me to set the indentation of the chunks when they're used, not when they're written
3. Still lets me work in Emacs

Thanks!

---

Correction: noweb does allow me to indent later --- I misread the paper I found on it.

By default, notangle preserves whitespace and maintains indentation when expanding chunks. It can therefore be used with languages like Miranda and Haskell, in which indentation is significant

That leaves me with only the "Runs on Windows" problem.

Solution 1

I did this:

http://sourceforge.net/projects/pywebtool/

You can get any number of web/weave products that will help you construct a document and code in one swoop.

You can -- pretty easily -- write your own. It's not rocket science to yank the Python code blocks out of RST source and assemble it. Indeed, I suggest you write your own Docutils directives to assemble the Python code from an RST source document.

You run the RST through docutils rst2html (or Sphinx) to produce your final HTML report.

You run your own utility on the same RST source to extract the Python code blocks and produce the final modules.

Solution 2

I have written Pweave http://mpastell.com/pweave, that is aimed for dynamic report generation and uses noweb syntax. It is a pure python script so it also runs on Windows. It doesn't fix your indent problem, but maybe you can modify it for that, the code is really quite simple.

Solution 3

The de-facto standard in the community is IPython notebooks.

Excellent example in which Peter Norvig demonstrates algorithms to solve the Travelling Salesman Problem: https://nbviewer.org/url/norvig.com/ipython/TSP.ipynb

More examples listed at https://github.com/jupyter/jupyter/wiki

Solution 4

You could use org-mode and babel-tangle.

That works quite well, since you can give :noweb-ref to source blocks.

Here’s a minimal example: Activate org-babel-tangle, then put this into the file noweb-test.org:

#+begin_src python :exports none :noweb-ref c
abc = "abc"
#+end_src

#+begin_src python :noweb yes :tangle noweb-test.py
def x():
  <<c>>
  return abc

print(x())
#+end_src

You can also use properties of headlines for giving the noweb-ref. It can then even automatically concatenate several source blocks into one noweb reference.

Add :results output to the #+begin_src line of the second block to see the print results under that block when you hit C-c C-c in the block.

Solution 5

You might find noweb 3 easier to build on Windows. It was designed to be more portable than standard noweb.

View more solutions

Comments

[jkp]

I think point two is never going to be easy to solve: indentation is key to the Python language so my guess is you will never solve that one.

[user1066101]

@jkp: Actually, it's relatively easy to resolve the indendentation when emitting the final source distinct from the literate programming document. As long as your LP language imposes some rational rules on what kind of Python statement suites it will work with.

[0 _]

Apparently someone has implemented the python equivalent: noweb.py.

[Sherwood Botsford]

Why was this question closed? Seems to me that any question that gets this many upvotes for the question, 40 votes for answers is answering a need. Revisit the rules?

[JasonFruit]

@Sherwood Botsford you must be new here.

[JasonFruit]

I'm looking through your approach --- it looks pretty close to what I'd like to have. I'd much rather adapt myself to someone else's tool than write one of my own; look what's happened with web frameworks!

[JasonFruit]

Everything about this looks good, except that when I generate LaTeX, there's a bunch of nuweb markup that MiKTeX and TeTeX don't seem to understand; I can't find a way to remedy that. Can you explain how it works? I read your excellent documents, but I'm not sure what you mean when you say, "The biggest gap in the LATEX support is a complete lack of understanding of the original markup in nuweb, and the very real damage done to that markup when creating pyWeb."

[user1066101]

@JsonFruit: Without understanding any LaTeX, I converted some parts of nuweb. Since I didn't take the time to understand any of the LaTeX, I suspected it was wrong. You've found out for a fact that I converted it wrong. Since the LaTeX markup is relatively easy to isolate, you might want to consider fixing the Markup. Or use nuweb (or noweb) directly -- they might be better.

[JasonFruit]

I'll play with the LaTeX a bit - the parsing code is simple and easy to read. If I come up with a satisfying solution, I'll send you some patches.

[JasonFruit]

I'll check it out once the site is responding again.

[JasonFruit]

I gave it a look --- it seems much more likely to work for me than the previous version. I quite likely would have gone with this had I realized it was available before I settled on another tool.

[naught101]

Ipython/Jupyter Notebooks are good for exploration, but they really don't compare to rmarkdown/Sweave - they are painful to work with in git, because they store a lot of metadata and output in the XML file that's not necessary to keep a history of (really, you only want the generators, not the output). Pweave is a better solution for literate programming.

[Baum mit Augen]

In this case it's not your fault because the question was off-topic, but please note that in general, an answer that recommends some tool or library must demonstrate how it solves the problem at hand in the answer itself.

[alpha_989]

@Colonel-panic and all Thanks for the point this out..Couple of questions...

[alpha_989]

Is there a way to extract out the python code into a separate python file in Jupyter notebooks?

[alpha_989]

Can I run a jupyter notebook at the commend line, such that it will execute the entire code and capture the resulted generated images/other ouput into the jupyter notebook?

[Sherwood Botsford]

Please edit your answer to have a current link. The present (2021-Jan-6) one has a terse description and no source.

[akaihola]

There's Jupytext for Jupyter.