ContourViz: Creating Interactive Contour Visualizations from the Command Line

Despite the author name, this is a guest post from Christopher Witulski; he can be reached at  chris.witulski at gmail.com.  We thank him for sharing this exciting pre-publication work. -- MSC

Last year I learned about music21 and ever since I have been wondering how I can use it to learn more about the Moroccan musical repertoires that I study. Long story short, I ended up building a tool for creating interactive web-based contour visualizations from the command line and I'd like to share it here.

Climbing out of a rabbit hole

I was working through a project and struggling to keep track of things. The paper was an analysis of a genre of Moroccan sung poetry called malhun for the 2016 Analytical Approaches to World Music conference. The performance of each poem can last twenty-ish minutes and contains a number of repetitions of the refrain text. These refrains are short (roughly eight 2/4 measures long) and modulate through repetitive--but different--melodies. I had transcribed over sixty of them in an attempt to understand how they worked, how they changed, and how they were related to each other.

Malhun Performance in Fez, Morocco

Having performed this music in Morocco (I'm the lone violinist in the photo who can't quite figure out how to play while holding the instrument upright on my knee), I was constantly struck with this feeling of déjà vu. New melodies felt so similar to old ones, but I could not put my finger on how or why. The problem was simple: I could not keep sixty or seventy different transcriptions in my head at once. Comparing them was getting tricky. I wanted a way to stack them on top of each other, almost as if I could print the transcriptions on a transparency and show them all at once on an overhead projector.

Over the previous months, I had been teaching myself Python in an effort to learn more about music21 and what it could do. It was time to try and build the tool that I needed instead of wishing I could find it.

Visualizing contours

For the presentation, I put together a small library that carried out two main tasks. First, it used music21 to parse my transcriptions, normalize the length of each melody, and build a dataset. Using offsets, frequencies, and distances from the final note of each melody, it turned note objects into a JSON of coordinates. At 1,000 different y values (each corresponding with 1/1,000th of the length of the total melody), it measured an x value for the frequency and one for the "distance from the root," the distance in steps above or below the melody's final pitch.

Malhun Contour Example

The JSON was passed to another library that I had been recently learning and working with called D3.js. It is written in JavaScript and designed for creating powerful interactive data visualizations. I supplemented my presentation with an online chart of each of my malhun transcriptions: by grouping contour lines within each poem, I was able to easily see the source of my déjà vu. Despite changes in pitch content, range, root motion, and a host of other things, the contours themselves often stayed strikingly consistent throughout the long performances. You can see the visualization and click through the different poems online, though be aware that some parts (like the "Next" button) are artifacts of the paper presentation.

Building a tool

Maybe two weeks ago I decided to try my hand at creating a Python library of my own. I simplified the chart, creating a sort of template, remove the stepwise element of the visualization, and fought my way through learning to upload a project to PyPI. The result is ContourViz... I didn't give much thought to the name, my apologies.

ContourViz, simple example

The tool takes, as an argument, either a music notation file or a directory with many of them. It parses these files and creates a JSON structure of 1,000 coordinates for D3.js to work with. It then copies a folder called results that includes an index.html file and a folder of JavaScript and CSS files that the generated web page will use into the current directory. Finally, it runs the Python SimpleHTTPServer and opens the new page, parsing the JSON to create the visualization.

You can install ContourViz using the following in your terminal:

pip install contourviz

It runs from the command line, so creating a visualization of multiple melodies, like the one above, is as easy as:

chart-single-contour '/path/to/file.xml'

Working with a directory is similar:

chart-contours '/path/to/directory/full/of/xml/or/mxl/files'

ContourViz, more complex example

I'm still toying with the system and it has a number of issues. For example, I would love for it to parse voices as individual melodies if they are present. Instead, it only works with monophonic lines, meaning that each voice has to be in an individual file if you wanted to visualize voice leaning or other contrapuntal patterns. There are smaller issues: I still need to set up the Y axis to render note names properly.

Please feel free to check out the GitHub repo and suggest any other changes or ways in which it could be more helpful. This is my first go around at building a tool of this sort, so I am eager to hear if it is helpful and how it could be improved. And thank you for allowing me to join the community.

Using music21 for Jazz Improv and Study

The following is a guest post from Daniel McGillicuddy, alias Basso Ridiculoso.  He can be reached at daniel.mcg [at] gmail.com.   -- MSC

Hello all!

I am a gigging musician and bass player who has discovered music21, but, alas, I am certainly not a musicologist or academic.

I have seen many of the amazing examples that showcase music21’s capabilities with classical and twentieth-century music, and wanted to show how I use music21. Hopefully these examples show that music21 can also be used to explore jazz and popular music, either via analysis for educational purposes or for developing improvisational ideas.

Jazz Standard Voice Leading Lines

Music21 has an amazing corpus of public domain classical music, but most jazz standards are not available for inclusion. But, since music21 has an understanding of seventh chords and reads MusicXML, a virtual corpus of jazz standards is available for analysis and exploration via another application called IRealPro. IRealPro is a virtual accompanist software program that has chord charts for over 3000 jazz standards, and which can export the chord progressions in MusicXML, a format that will allow music21 to understand the harmony. Once we have that outline of a jazz standard's harmonic structure, music21 can be turned loose.

For this example, lets export the chord chart to the standard “Alone Together” and generate a 3rd to 7th voice-leading line through the entire tune, based on this concept by Burt Ligon, as described here.

(links: Alone Together.XML and Guide Tone Lines with Music21.py)

Since music21 understands harmony, any kind of voice leading line is possible, for instance the 5th resolving to the 9th. Now these voice leading lines can be generated for any jazz standard (or for any chord progression) that can be exported as MusicXML format and these lines can be used as jumping off points for making solos or studying voice leading.

Jazz Solo Analysis 

Analyzing jazz solos from the masters is another way to get improvisational material, but it is better known as stealing someones licks! Since music21 can understand the relationship of any note to any chord, it can be used to analyze the functional relationship of the notes in a solo.

Here is an example of Miles Davis’s solo on “Freddie Freeloader” with the notes being labeled so they represent their function against the chord being played, for example, an F note on a Bb7 chord being the fifth.

(links: Miles Solo XML  and Melodic Labeler.py)

This same Music21 code was used to analyze Charlie Parker's solo on Bloomdido, and a walking bass line over F blues by Ron Carter.

Now any solo line that can be exported as MusicXML can be analyzed by music21 and then explored even further. What notes are favored? What beats of the bar do certain notes get played on? How many times do certain notes get played? Are there repeating phrases that a certain player uses over and over? All of this can be cataloged or graphed once it has been brought into the music21 world. The included code needs a chord symbol over every measure.

Hopefully these examples show that music21 is not only for musicologists exploring the pitch class space of Bartok's string quartets or for twelve-tone row composers! Students and musicians can use it for very useful and practical purposes as well. Many thanks to Michael for allowing this guest posting from big music21 fan!

(Ed: Thanks Dan! The examples included here are copyrighted by their respective composers and publishers. We believe their inclusion here for educational and instructional purposes are supported by all four factors of the Fair Use test).

"Data in a Major Key": MIT Spectrum, FSU

MIT Spectrum has an article by Kathryn M. O'Neill on my work, music21, and computational musicology:

“IF I WANT TO KNOW how the guitar and saxophone became the important instruments throughout classical repertory or how chord progressions have changed, those are questions musicology has been unable to approach,” says Associate Professor of Music Michael Cuthbert. Spotting trends and patterns in a large corpus of music is nearly impossible using traditional methods of study, because it requires the slow process of examining pieces one by one. What his field needed, Cuthbert determined, was a way to “listen faster.”

Read more at http://spectrum.mit.edu/articles/data-in-a-major-key/.

In other news, Clifton Callender at Florida State University is currently teaching a doctoral seminar on music theory techniques using music21.  His course description is at http://cliftoncallender.com/teaching/.

Music21 v. 2. (non-beta) released!

The long-awaited (at least by me) version 2 of music21 is released!  This is the first version of the v.2 release to be out of beta and stable enough for general use by everyone.

Upgrade with:

    pip install --upgrade music21

Or download from GitHub.

The first non-beta release of music21 since v. 1.9.3 (June 2014) gives a ton of new features and lots of new speed.  But being a major release change number, it also has some changes that every programmer using the system needs to be aware of.  The release notes on GitHub gives all the details, but here are the highlights since 1.9:

Changed and Added features

* Duration and Offset now use Fractions when necessary for exact representation of tuplets. Many, many errors from rounding are gone.  For now, you can use Duration.quarterLengthFloat and offsetFloat to get the old behavior, but float(Duration.quarterLength) and float(offset) are better.
* Converters support easy to install custom sub converters. MEI is now supported (thanks to McGill university)
* Python 2.6 is not supported.  Python 3.4 is highly recommended; 2.7, 3.3, and 3.5 also work.
* Loading cached streams is extremely fast. All streams are automatically cached when loaded from disk.
* Sorting is much more consistent and faster
* MusicXML parsing and showing have been rewritten to use cElementTree and many new features.
* Stream's internal mechanisms have been hugely rearranged.  Now offsets are stored inside Streams instead of inside Notes, etc., making lots of things faster and more reliable.
* Streams support filters on iteration using the `.iter` property and the `recurse()` method.  These are big changes for speed and reliability.
* Namedtuples replace anonymous tuples in many places
* Music21 is available under the BSD license.
* Musedata files are no longer available in the corpus. However, new files in MusicXML format have replaced several of them.
* Complete rewrite of TinyNotation making it much easier to subclass for your needs.
* If you have MuseScore 2, try sc.show('musicxml.png') to get a beautifully rendered musicxml file. Or use .pdf to get something ready to print.  Thanks Nicholas, Thomas, and Walter!
* Builds are automatically tested for errors and documentation coverage.
* Experimental modules moved to the `alpha` sub package.  `demos` reorganized.
* Lots of documentation changes!
* Obscure and almost never used (or actually never used) methods and attributes have been removed.
* Did I mention how much better the documentation is getting?

In case anyone is keeping track, since v.1.0 (June 2012), here are the:

Biggest changes between 1.0 and 1.9

* Store complete Streams via FreezeThaw
* Output to Vexflow and `music21j`
* Converters have been moved into packages.
* It takes 1/3 the time to do most operations, and 1/4 the time to start up.
* Capella supported.  ABC imports almost everything. Humdrum supports multiple voices. Chords have a better root() algorithm
* Many, many new corpus pieces.
* Layout support.
* Python 3 supported, and now recommended.
* Timespans make .getContextByClass at least an order of magnitude faster, letting music21 handle huge scores.
* Derivations reduce the number of Streams to keep track of.


Oh, and I did more than patch bugs in the last week:

Release notes since 2.0.11

* Streams use .iter and .recurse() in TONS of functions, making many a lot faster, a few a bit slower, but all cleaner to debug and safer.
* Deprecated items now return a deprecation warning.
* Duration objects now have a `.client` which can inform the `Note` of changes to it.
* `.classes` searches are way faster. Returns tuple.
* `deepcopy` is about 30% faster.
* `common` is split into a directory of related functions.  Now worth looking through.
* all corpus files, including small .abc files with non-standard additions, now parse.  A complete corpus.search().parse() should be possible without any try: statements.
* several bugs in musicxml processing (mainly related to the handling of expressions, noteheads, etc., on chords) have been fixed.  Also Finale's `` tag is supported.
* code is much more "lint-free" catching many subtle bugs.
* audioSearch is cleaned up, with beta-type code moved to demos.
* Documentation much improved including three new User's Guide sections, and (thanks to bagratte) fixes for UTF-8 errors.
* `io.open` replaces `codecs.open` for better non-Western script handling.
* .egg files are no longer distributed.  I'll work on getting .whl (wheel) files soon, but for now use .tar.gz.  PyPi no longer supports .egg, so there's no reason for them.

 incompatible changes

* `.fullyQualifiedClasses` is GONE. No one used it.  Instead a new `.classSet` replaces it for rapid class searching.
* sites.Sites and sites.SiteRef are no longer imported into base by default.
* `documentation` modules reorganized, with better examples.
* `stream.core` moves several core modules out of the `stream` module.
* `Volume.parent` renamed `Volume.client` to match `Derivation` and `Duration`
* `.components` on `Duration` now returns a tuple.

What's Next?

Today also announces the first commit of music21 3.0 -- for the first time, I'm going to try to do something daring: keep bug fixes and some backwards compatible changes in the 2.1 (2.2, etc.) branch, but go forward with bigger changes in a 3.0-alpha branch.  Some things that you might expect to happen:

* All deprecated functions will be gone in 3.0; like immediately; like I'm deleting them as I type.

* Lots of things that currently return a Stream will instead be iterators over Streams.  These include: .getElementsByClass(),  getElementsByOffset() -- the fact that so many streams get created is one of the biggest headaches and reasons why the system gets slow.  You can prepare for the change by examining your usage of these functions and asking yourself, "Am I actually using this as a Stream? Or just as a bunch of objects to iterate over in a for loop or to count using len()"?  If the latter, you're fine.  If the former, go ahead and add .stream() after it, for instance filteredStream = s.getElementsByClass("TimeSignature").stream().  The last `.stream()` call does NOTHING right now, but it will ensure that your code works exactly the same after the change happens.  If you want to use the new features (even in 2.1) add .iter between `s` and `.getElementsByClass()` (but leave off the `.stream()`.  You'll find that life will be going a lot lot lot faster.

* I'm going to make a second attempt to use TimeSpans as a general storage engine for Streams.  These are the super fast representations of Streams that Josiah Oberholtzer made, that speed up working with large streams by 10-100x. But for very small streams (such as one measure of a Chorale), they are much slower than the current Streams. Now that all the core mechanisms are factored out of Stream into StreamCore, I can play much more easily with switching in any out the backend functions. Using the lessons of Python's TimSort, I'll probably have the TimeSpan core kick in immediately when there are more than 64 elements in a Stream; it should be seamless except for a tiny delay when the 65th element is added (like shifting gears in a car).

* I may make Python 3.4 a requirement.  We'll see... I'm sick of coding for Python 2.  Python 3 is much more fun from the coder's perspective.

Thanks everyone for great support! -- Myke

Music21 v. 2.0.11 released

Ten days since the last release, so time for a new one. Again, speed, stability, and new features. The biggest change is the entirely new MusicXML output system to match the entirely new input system introduced last release. 

The second biggest is in the (re)introduction of StreamIterators and RecursiveIterator. I'll need to get some demos up of this soon, but this will be a game changer for some tasks.

Update or install with one (or both) of these commands: 

pip install --upgrade music21

pip3 install --upgrade music21

Bigger changes

1. MusicXML now uses the faster, more reliable ElementTree output generator. Please report any bugs on import or export, especially if they are regressions from format='oldmusicxml'. oldmusicxmlwill disappear soon.
2. Better docs (see below), especially for the long under documented recurse function. Everything that was in overview is now in the User's Guide.
3. Streams now support filters on iteration -- if you have been using: for e in s.getElementsByClass('X'), try: for e in s.iter.getElementsByClass('X') for a major speedup, especially if you just want the first one or something of that sort. Recurse() supports the same, so for e in s.recurse().notes.getElementsByGroup('tuba') will be WAY faster than before. You might not notice the difference on your own work, but internally things are getting a lot faster. (obscure non-filter routines will be deprecated and disappear soon).
4. Corpus docs/indexes, etc. are updated with more recent corpus changes (nothing new, but easier to find).
5. Use of deprecated functions now generates a warning. This should help people plan for migration in case you're not reading the documentation religiously.

Smaller changes

1. Documentation is improved and updated working with Jupyter/IPython 4 (note: a bug in nbconvert + pandoc requires pandoc v. 1.33 or older to make; they're working on a patch). Docs build in parallel, so it's very fast -- you'll see updates more often.
2. Documentation is now separated into "source/" and "autogenerated/" folders -- everything in source is user editable. Nothing in autogenerated is.
3. A number of obscure, long deprecated functions are gone, the biggest being n.removeLocationBySite() use n.sites.remove()
4. normalization in features has been fixed (Thanks Frank Zalkow)
5. Parsing of cappella MusicXML files has been improved.
6. Improved parsing of RomanText files; bugs in several encodings of rntxt and abc files have been fixed.
7. common.nearestCommonFraction has been renamed addFloatPrecision to better reflect what it does. This has always confused me.