Harvard FAS Informatics Python Intensive Workshop

Developer notes

This workshop is in the form of Jupyter notebooks (.ipynb). When running the workshop, students run the notebook on Google colab, ensuring easy setup and equivalent environments for all students. We also convert and host these notebooks on our website. In order to have hidden and expandable "Solution" code blocks for exercises in both contexts, there are a few things we need to do while editing the notebooks. These should require minimal effort on our part.

When editing a notebook (Python-PartN.ipynb)

⚠️ Important: You only ever need to edit the regular notebooks (Python-PartN.ipynb)! Do not edit *-student.ipynb files or *.md files directly! Your edits will be overwritten. See below for explanations of these files.

Coding exercises

1. All coding exercises must have two associated code blocks: a student block, in the state the students should see the exercise before starting, and a solution block, with the final, working example coded.
2. Solution blocks MUST begin with the following comment: #@title Solution {display-mode: "form"}. This allows the block to be hidden in Google colab and by our website.

Tagging exercise blocks that would throw an error

Because:

1. We purposefully have some code blocks that will throw errors when run, for demonstrative purposes, and
2. We leave some code in student exercise blocks as a scaffold that would throw an error and leave unwanted output in our website markdowns when executed,

exercise blocks with scaffold code that would show an error MUST be tagged as no-execute. This lets the markdown converter know not to execute them (and output their error text) during conversion.

Tags can easily be added to a Jupyter notebook cell in VSCode by selecting the code block, clicking ... in the menu that appears in the upper left corner of the codeblock, and selecting "Add Cell Tag". A dialogue window will pop-up on the top of the VSCode window where you can enter the tag. Type "no-execute" and hit enter. The tag should now appear at the bottom of the code block.

Non-coding exercises

For non-coding exercises, wherever applicable hide the solution in <details></details> tags. This HTML is rendered as a hidden and expandable block in both in Google colab and by our website. For example:

<details><summary>Solution</summary>
This is the solution to the exercise!
</details>

Will render as:

Solution

This is the solution to the exercise!

Avoid template syntax

Our website renders these pages with the mkdocs-macros plugin, which treats two opening curly braces in a row, or an opening curly brace followed by % or #, as template syntax. Don't use these character combinations anywhere in a notebook (text, code, or output), or the website build will fail.

But wait, why is there still a -student.ipynb file?

Isn't the point of all this to maintain only one notebook?? Correct.

The -student.ipynb files are generated by the .github/scripts/jupyter_convert.py script. These are identical to the regular .ipynb files, but with all rendered output stripped. We want to ensure we distribute this version to the students so they get a clean notebook after we may have edited and run some code in them.

All changes made to the regular notebook files are reflected in their corresponding -student.ipynb file when the GitHub Actions workflow runs (see below).

And why does each notebook have an associated markdown file?

The markdown files (.md) are also generated by .github/scripts/jupyter_convert.py. These are the files that we link to on our website and contain formatting for that context.

How the .md and -student.ipynb files are generated

The GitHub Actions workflow in .github/workflows/nbconvert.yml runs .github/scripts/jupyter_convert.py and commits the regenerated .md and -student.ipynb files. It runs on any push that changes a notebook in the root of this repository (other than the -student.ipynb files) or .github/scripts/jupyter_convert.py. It can also be started manually from the Actions tab of the repository (Run workflow).

⚠️ Note: The workflow commits the regenerated files back to the branch you pushed to, so pull after pushing before making more changes. Each run re-executes every notebook, so you may also see small changes to .md files of notebooks you didn't edit (e.g. timestamps in download output).

You can also generate these files locally to preview them (see Running the conversion locally).

⚠️ TL;DR: You never have to edit the -student.ipynb or .md files by hand. They are generated from the regular notebook files by .github/scripts/jupyter_convert.py. In fact, never edit these files directly since your edits will be overwritten.

What if I edit a notebook to add an image or change the order in which the images appear?

We add alt-text to the images automatically in .github/scripts/jupyter_convert.py. Unfortunately, since some of these images are programmatically generated (e.g. plots from the seaborn section), the best solution I could come up with was to hardcode in the alt-text strings in the order in which they appear in the document and add them in sequence upon conversion to markdown.

This means if you ever add an image or re-order the images, .github/scripts/jupyter_convert.py will have to be updated to reflect the new image/image order. The script treats every line of the generated markdown that starts with ![ as an image, including images in markdown cells, and stops with an error if there are more images than alt-text strings.

Running the conversion locally

From the root of this repository, run:

python .github/scripts/jupyter_convert.py

This executes and converts every notebook listed in the jupyter_files dictionary in the script. To convert only specific notebooks, give their file names as arguments:

python .github/scripts/jupyter_convert.py python-data-validation.ipynb

The script exits with an error if a given file name is not in jupyter_files.

Running the script requires the same packages that the workflow installs (jupyter nbconvert pandas seaborn scikit-learn) and network access, since the notebooks download their data. The data files downloaded by the notebooks (e.g. bird_names.csv) are saved in the root of the repository. Don't commit these, since the original copies are in data/.

The script does not stop if a code cell raises an error, since some cells raise errors on purpose. Check the generated .md files for unexpected error output.

Adding a new notebook

1. Add an entry for the notebook to the jupyter_files dictionary in .github/scripts/jupyter_convert.py, with its title, description, authors, and alts. The title and description are written into the YAML front matter of the .md file, so they can't contain double quotes. alts needs one alt-text string per image, in order (see above). Use an empty list if there are no images.
2. Add links to the Colab notebook (the -student.ipynb file) and the .md file to index.md.
3. Commit and push the notebook, index.md, and .github/scripts/jupyter_convert.py. The workflow will generate and commit the .md and -student.ipynb files. To preview them first, run the conversion for the new notebook locally (see Running the conversion locally).