Remote musical MIDI collaboration has been an interesting academic research area for years, but has not been explored by many casual musicians. One reason is that the complexity of software that brings MIDI and Networking together makes it a little bit of a daunting endeavor. We think it’s time to open exploration to more people and make remote MIDI collaboration as easy as joining a Hangout.
What is Network MIDI?
Back in 2001, a group of researchers at Berkeley began to experiment with remote musical collaboration [1]. The idea was see if musicians separated by some distance could collaborate in real time over a high-speed network. Rather than sending real-time audio signals, MIDI events were transmitted between instruments at two different locations.
To validate the idea, experiments were performed with musicians at Stanford and Berkeley who were “jamming” in real-time with MIDI instruments. The MIDI events from the studio at Stanford were sent to Berkeley, and the MIDI events from the Berkeley studio sent to Stanford. The researchers had access to a special high-speed network between the two locations which offered much better performance than was available to the general public in 2001.
The researchers discovered that there were inherent problems in using internet technology to transmit realtime MIDI note information: These were loss, latency and jitter. Loss occurs when packets disappear or arrive out of sequence during transit over the network. Latency is the delay that is inherent in the network. Jitter is the unpredictable variation in latency as network messages encounter unpredictable buffering.
These effects could be heard in a distributed MIDI session. Musically, Loss manifests itself as notes that remain “stuck-on”, or a sustain pedal that does not release. Latency is the delay between when a key is depressed and when a note sounds. Jitter is unpredictability in Latency. Musicians can learn to compensate for constant latency of up to 20 or 30 mS [3]. Church organ players compensate for latency routinely. Jitter is much harder (or impossible) for a musician to compensate for.
MIDI information can be sent directly over a network connection by either TCP or UDP protocols. A TCP connection provides guaranteed delivery of a byte stream, but at a potentially high delay (or latency). This makes TCP generally unsuited for real-time delivery of note-ON/note-OFF MIDI events. UDP can be used to send short messages and with much lower latency. This makes UDP better suited for real-time sending of note-ON/note-OFF events. But UDP makes no guarantee about delivery, and so UDP streams inherently suffer from Loss.
To counter the effects of Loss inherent in UDP, but to leverage its latency characteristics, Lazzaro and Wawryznek developed a new protocol for sending MIDI events of a network. This protocol was described in RFC-4695, and later in RFC-6295. The protocol has two common names: “Network MIDI” or “RTP MIDI”, where RTP stands for “Real-time Protocol.” The Network MIDI protocol builds on simple packet transmission to add extra information about the state of a MIDI system distributed across two computer nodes. In this system, each packet sent includes extra information about the packets that preceded it so that a receiver can determine if some previous messages were lost.
For example, when a “note-ON” is transmitted, it might include information about some prior “note-OFF” or “damper pedal release” events. If the receiver calculates that it missed a “note-OFF” packet it can quiet the still-sounding note. When a receiver software corrects for lost information, it is “recovering” missing information. This helps the musical performance to sound pleasing, even when information has strictly been lost.
Latency and Jitter are not strictly accounted for in RTP-MIDI. Common network techniques can be used with the recovered MIDI stream to handle jitter and to make latency a constant. (A Jitter Buffer is a common technique.) These techniques have the unfortunate effect of adding latency, however.
RTP-MIDI Drivers
RTP-MIDI has been built into Apple OSX since approximately 2004, and later on it was included in IOS. Mac musicians are pretty used to connecting an iPad up to their MacBook wirelessly using Network MIDI. For the PC, there is a freely available rtpMIDI driver from Tobias Erichsen [2]. And of course, McLaren Labs provides one for the Raspberry Pi and Ubuntu.
A MIDI BRIDGE
Remote musical MIDI collaboration has been an interesting academic research area for years, but has not been explored by many amateur musicians. The complexity of software that brings MIDI and Networking together makes it a little bit of a daunting endeavor. We think it’s time to open exploration to more people.
McLaren Labs wants to make it easier to experiment with sending real-time MIDI over the internet by introducing the MIDI Bridge. Like a video-conference bridge for voice and chats, the MIDI Bridge lives in the cloud and routes MIDI events between two participants. The figure below shows two participants engaged in a MIDI sharing session using the MIDI Bridge. Studio1 connects to the Bridge, and Studio2 also connects to the Bridge. The Bridge sends the events from Studio1 to Studio2, and the events from Studio2 to Studio1.
Connecting to the Bridge is simpler than connecting Studio1 directly to Studio2 because there are no complicated firewall rules to set up, and the IP address of the Bridge is easily discovered by both Studio1 and Studio2. So setting up a session just involves both studios calling into the same bridge.
The process of “calling into” can be done using McLaren Labs rtpmidi, or the MIDI Network Setup built into OSX.
Local-Mix or Bridge-Mix
The MIDI Bridge can be operated in one of two modes. Each participant can receive only the other participant’s notes, or each can receive the mix of the participants.
In Local-Mix Mode, the Bridge assumes that each participant sounds his notes locally. The input stream from one studio is forwarded to the other studio, and vice versa. The performance that a musician hears is the combination of his own local performance, and the delayed performance of the remote participant.
In Bridge-Mix Mode, the Bridge sends each participant the mix of the event streams of the participants. In this mode, Studio1 would not sound his notes as he plays, but would hear his events as they are sent back to him from the bridge. This lets Studio1 hear his notes as they sound transmitted over the network. A musician in Studio1 might be able to learn to compensate for his delay and might be able to use this experience to gauge how his performance is being perceived in Studio2. Likewise, Studio2 hears his notes as they sound modulated by the network.
At the present time, the MIDI-Bridge does not offer latency or jitter protection. It does implement loss recovery for Note-ON/OFF, Control Change, Program Change and Damper Pedal. Performers will hear network latency modulated directly onto their event stream. This might be a plus for some types of experimentation.
Availability
The McLaren Labs MIDI Bridge is available for use as of June 2020. Send us a note at info@mclarenlabs.com to receive an invitation.
What you need
- an ethernet connection from you computer to your router (it is best to avoid WiFi)
- an external USB keyboard or instrument for sending note information
- a DAW or synth or external synth for playing local and remote notes
Summary
Let’s see what people can come up with when joining a Network MIDI session over the Internet is as easy as a joining a Hangout!
References:
This is the article that inspired our work in Network Musical Performance.
[1] John Lazzaro and John Wawrzynek (2001). A Case for Network Musical Performance. The 11th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2001) June 25-26, 2001, Port Jefferson, New York
[2] Tobias Erichsen – provider of a free RTP-MIDI driver for PC
https://www.tobias-erichsen.de/software.html
Some excellent articles on remote musical collaboration.
[3] “An Overview on Networked Music PerformanceTechnologies”
Cristina Rottondi, Chris Chafe, Claudio Allocchio, Augusto Sarti
https://www.researchgate.net/publication/311447746_An_Overview_on_Networked_Music_Performance_Technologies
“Towards the Internet of Musical Things”
Luca Turchet, Carlo Fischione, Mathieu Barthet
http://smc2017.aalto.fi/media/materials/proceedings/SMC17_p13.pdf