This example demonstrates how Strasheela can be used in realtime — interoperating with SuperCollider. Supercollider algorithmically generates a single voice using Supercollider patterns. The notes of this voice are send to Strasheela via Open Sound Control (OSC). Strasheela considers this voice the cantus firmus and creates a fitting counterpoint for it in realtime. Strasheela sends the counterpoint back to Supercollider (again via OSC), and Supercollider plays both voices synchronously (with a small latency).
The example implements a very simple variant of first species counterpoint. The counterpoint is homophonic to the cantus firmus. However, in this example the cantus firmus is rhythmically free. The CSP implements the following rules for the second voice.
The core idea of the realtime constraint solver demonstrated by this example is very simple. It conducts a search like a normal Oz solver. However, it is given a maximum search time. If no solution is found after this maximum search time (or if the search failed), then a user-defined default solution is returned. That way, the solver is never busy for too long and always responses to new input. In this example, the default solution is nil, and in case of a timeout or fail, a note is simply omitted.
The maximum search time is compensated for by a latency which delays all notes of the example (i.e. the cantus firmus generated by Supercollider and the counterpoint from Strasheela). If the maximum search time and the latency is high enough, the search should never time out nor fail in this example. No dropped note (no timeout) nor any late note were produced by this example with 10 msec maximum search time and 35 msec latency (for OSC network traffic) on a Macbook Pro 2.2GHz.
Please note that the code examples below are partly Strasheela code and partly Supercollider code. However, the language is always indicated. The full implementation of the example presented here is also available as source, split into two source files: one Supercollider source file and one Strasheela source file. The explanation below is quite detailed: you can see it as a mini tutorial how realtime constraint programming can be done with Strasheela.
This section explains how the communication between Strasheela and Supercollider is set up. We will first look at Strasheela, and then at Supercollider. In later sections, we will customise the definitions of the present section to specify what Strasheela and Supercollider actually do when receiving OSC in this example.
Strasheela currently uses the UNIX applications sendOSC and dumpOSC (available here) for its OSC interface. Hence, the OSC interface (and also this example) is only supported on UNIX systems (e.g., Linux or MacOS X). Firstly, we must load the Strasheela functors required for realtime constraint programming. ModuleLink should be defined in your Oz initialisation file (cf. ../_ozrc), and you may add the following code to this file as well. Please remember that variables (e.g., OSC and RT) must be declared first (e.g., by preceding the keyword declare at the beginning of your Oz initialisation file).
[OSC RT] = {ModuleLink ['x-ozlib://anders/strasheela/OSC/OSC.ozf'
'x-ozlib://anders/strasheela/Realtime/Realtime.ozf']}
The following four lines set up OSC input at port 7777 and output to port 57120 (the port of sclang, the Supercollider language application).1
OutPort = 57120 % sclang port InPort = 7777 MySendOSC = {New OSC.sendOSC init(port:OutPort)} MyDumpOSC = {New OSC.dumpOSC init(port:InPort)}
We can test the Strasheela OSC interface by setting OutPort and InPort to the same number, so that Strasheela sends OSC packets to itself.
OutPort = InPort = 1234
MySendOSC = {New OSC.sendOSC init(port:OutPort)}
MyDumpOSC = {New OSC.dumpOSC init(port:InPort)}
The following code line will cause Strasheela to browse all OSC package received at port InPort. The method getOSCs returns a stream of packages (i.e. a list whose tail is unbound and which can therefore be extended).
{Browse {MyDumpOSC getOSCs($)}}
Instead, the next line installs an OSC responder which reacts only to messages with the address pattern '/note'. Whenever a message with this address is received, the corresponding procedure is called. Here, the procedure also just browses the message. The responder also receives a timetag, which is 1 in case a single message was sent. In case the message was sent within a bundle, then the timetag of the enclosing bundle is received. See below for details.
{MyDumpOSC setResponder('/note' proc {$ Timetag Msg} {Browse responder#Msg} end)}
OSC messages may contain strings, which are not properly displayed by the Browser by default (remember that strings are just lists of character, which in turn are just integers). You can customise the Browser either via its GUI (in menu "Options", entry "Representation..." tick Strings), or feed the following line.
{Browser.object option(representation strings:true)}
Now, we are ready to send test messages from MySendOSC to MyDumpOSC. The '\note' parameters of this example are only intended for testing, and have no further meaning.
{MySendOSC send(test(some "test message"))}
{MySendOSC send('/note'(0 1 2.0))}
Please note that OSC messages are expressed simply by Oz tuples for the send method, and the the received messages have the same format. The Strasheela interface also supports OSC bundles, including timetags. A bundle is a list, and the first bundle element is optionally a timetag. Bundles can be nested. Different timetag formats are supported, but the default is an integer denoting the milliseconds since midnight UTC of January 1, 1970. The following example sends a bundle with a timetag 1000 msec later than now and a single message. Please see the documentation for further details on the format of OSC packets in Strasheela.
{MySendOSC send([{OSC.timeNow} + 1000
'/note'(0 1 2.0)])}
We now come to the Supercollider side of our communication process. Supercollider already receives OSC per default at the port 57120. We only need to define what Supercollider should do when it receives OSC packets (see below). The following two lines define an interface for sending OSC.
~outPort = 7777;
~mySendOSC = NetAddr("localhost", ~outPort);
Assuming that OSC messages are still browsed on the Strasheela side, we can test the communication from Supercollider to Strasheela with the following code.
~mySendOSC.sendMsg('test', 'some', "test message");
~mySendOSC.sendMsg('/note', 0, 1, 2.0);
Finally, we test sending OSC from Strasheela to Supercollider. The following Supercollider code defines a function which is called whenever Supercollider receives any OSC message. This function simply prints any received message. However, once Supercollider's localhost synthesis server scsynth is started, it also sends OSC messages to the Supercollider language application. As we are not interested in the messages from the scsynth, the if conditional ensures that those messages do not cause any additional action.
thisProcess.recvOSCfunc = { arg time, addr, msg;
if ( addr.port != 57110, // ignore scsynth messages
{ msg.postln; })
};
We send OSC messages from Strasheela as before.
{MySendOSC send(test(some "test message"))}
{MySendOSC send('/note'(0 1 2.0))}
This section defines the sound playback in Supercollider. We define a very simple synthesis instrument: a saw oscillator which is filtered by a resonating low-pass filter. However, before we do that, we need to start Supercollider's localhost server.
s.boot;
The following code defines our 'synth' and sends it to the server. The synth expects the following optional parameters: the duration dur (measured in secs), the pitch freq (measured in Hz), the loudness amp (a float in the interval [0,1]), and the filter cutoff-frequency ffreq (also measured in Hz). If you don't know the Supercollider language, please refer to the Supercollider documentation for understanding this definition.
SynthDef("Strasheela-playback", { arg dur=1, freq=440, amp=0.3, ffreq=1000;
var env = EnvGen.kr(Env.perc(0.05, dur-0.05, 1, -2), 1.0, doneAction: 2);
Out.ar(0, RLPF.ar(Saw.ar(freq, amp*env), ffreq, 0.1))
}).send(s);
We test our synth "Strasheela-playback" with the following call.
Synth("Strasheela-playback", [\dur, 3, \freq, 72.midicps, \amp, 1, \ffreq, 2000]);
In the example, the two different voices will use different filter settings in order to make them better distinguishable.
We now define the actual constraint satisfaction problem. The following rules are defined in exactly the same way as rules for non-realtime CSPs. To keep the example simple, only very few rules are defined. Nevertheless, these rules make multiple score contexts interdependent (namely pairs of neighbouring melodic notes, and simultaneous notes), and that way result in a true search problem (although the problem is rather simple).
A harmonic rule — stating that simultaneous notes must be consonant — is implemented by two procedures. The procedure IsConsonance defines the actual rule, and the procedure GetInterval makes the definition of this and the rule more simple. GetInterval expects two notes and returns a fresh finite domain integer (FD int), constrained to the absolute distance between the pitches of these notes.
proc {GetInterval Note1 Note2 Interval} Interval = {FD.decl} {FD.distance {Note1 getPitch($)} {Note2 getPitch($)} '=:' Interval} end
IsConsonance constrains the interval between two simultaneous notes (a FD int) to a perfect or imperfect consonance up to an octave plus a major third at maximum. Note that the unison (Interval equals 0) is not permitted.
proc {IsConsonance Interval} Interval :: [3 4 7 8 9 12 15 16] end
The melodic rule RestrictMelodicInterval is very similar to IsConsonance: it expects the interval between two successive melodic notes (a FD int) and constrains it to either anything between a minor second and a fourth, or to a fifth, or to an octave. Again, the interval 0 (i.e. pitch repetition) is not permitted.
proc {RestrictMelodicInterval Interval} Interval :: [1#5 7 12] end
Finally, the rule IsDiatonic restricts the domain of a single note to pitch classes of the C major scale.
local ScalePCs = [0 2 4 5 7 9 11] % list of pitch classes in c-major scale in proc {IsDiatonic MyNote} {FD.modI {MyNote getPitch($)} 12} :: ScalePCs end end
In this example, Strasheela receives OSC packets from Supercollider and sends OSC packets back. However, internally Strasheela uses its own music representation. This section shows how single notes in both representations can be transformed into each other. For a CSP as simple as the present one this is not really required, we could instead use OSC messages directly. However, Strasheela's music representation provides a rich interface which can simplify the definition of more complex realtime CSPs with a more complex input or output score, and therefore such a transformation is worth showing. Moreover, using Strasheela's music representation we can also use Strasheela's score distribution strategies — which allow for a randomised variable value selection.
The transformations shown below depend on the variable Now. Now is bound to some value which serves as reference for the start time 0. The function OSC.timeNow returns the number of milliseconds since midnight UTC of January 1, 1970 (an int).
Now = {OSC.timeNow}
The function MakeScoreNote expects a Timetag and an OSC message representing a note. It returns a corresponding Strasheela note object. Supercollider sends the note parameters in the correct unit of measurement (temporal parameters are measured in msecs, Pitch is a MIDI key-number, and Amplitude is a MIDI velocity — all values are integers). Note that the note's start time is the Timetag minus Now (the Timetag itself is beyond the domain of a FD int). When a note is transformed back to OSC, Now is added to the note's start time (see below).
fun {MakeScoreNote Timetag '/note'(Duration Pitch Amplitude)} {Score.makeScore note(startTime: Timetag - Now duration: Duration pitch: Pitch amplitude: Amplitude timeUnit:msecs) unit} end
The corresponding function MakeOSCNote expects a Strasheela note object and returns an OSC bundle representing the note. The note's start time plus Now is the bundle's timetag. The note duration is transformed into seconds (a float) for more easy processing at the Supercollider side.
fun {MakeOSCNote MyNote} [{MyNote getStartTime($)} + Now '/note'({MyNote getDurationInSeconds($)} {MyNote getPitch($)} {MyNote getAmplitude($)})] end
This section finally shows the most interesting part of the example: how a constraint solver is called in realtime. We first define the constraint script, then create a constraint solver object for the script, and finally define an OSC responder which generates the next counterpoint note whenever it receives a cantus firmus note by calling the solver with its script again.
The structure of the following script definition is very similar to the scripts of non-realtime CSPs shown by other examples before. New is only the script argument Args, which is a record providing the script with multiple CSP parameters. The arguments of the procedure MyScript are its solution (here a single note NewNote) and a record of script parameters. We may call a script with two arguments an 'extended script'.
MyScript accesses two arguments from Args: the note which is simultaneous to NewNote and the melodic predecessor of NewNote. The simultaneous note is the note coming from Supercollider: the OSC responder hands this note to the scripts argument Args.inputScore (see below). The melodic predecessor note is the solution of the previous call of the CSP. Previous solutions (and previous input) are collected automatically by the solver and are made available in reverse order at the scripts argument Args.outputScores (respectively Args.inputScores).2
proc {MyScript Args NewNote} SimNote = Args.inputScore %% PrevNote can be nil (for first note and in case of no solution) PrevNote = Args.outputScores.1 % immediate predecessor of NewNote in NewNote = {Score.makeScore note(startTime:{SimNote getStartTime($)} duration:{SimNote getDuration($)} pitch:{FD.int 48#72} % MIDI key-number amplitude:64 % MIDI velocity timeUnit:msecs) unit} %% three simple rules {IsDiatonic NewNote} {IsConsonance {GetInterval SimNote NewNote}} if PrevNote \= nil then {RestrictMelodicInterval {GetInterval NewNote PrevNote}} end end
The rest of MyScript is strait forward. A note object is created and bound to NewNote and the rules shown before are applied to this note and its simultaneous note (or its melodic predecessor). Please observe that the constraint RestrictIntervalDomain is only applied in case the previous note is not nil (which is the case for the very first note and happens if the previous CSP call found no solution, see below).
As mentioned above, a maximum search time is given to the constraint solver. If no solution is found within that time, or if the search failed, then a default solution is returned. The following code defines the constraint solver object MySearcher, an instance of the class RT.scoreSearcherWithTimeout. The solver is an object, because it maintains an internal state between solver calls (e.g., the previous solutions). The solver is given the constraint script MyScript, a maximum search time of 30 msecs, and the default solution nil. Additionally, the list of initial output scores is specified as [nil] (i.e. the previous note of the very first note is nil), and the value selection of the score distribution is randomised.3 See the RT documentation for additional information on RT.scoreSearcherWithTimeout.
MySearcher = {New RT.scoreSearcherWithTimeout
init(MyScript
maxSearchTime:30 % in msec
defaultSolution:nil
outputScores:[nil]
distroArgs:unit(value:random))}
The following OSC responder reacts to OSC messages with the address pattern '/note'. Any received note message is first transformed into a Strasheela note using the function MakeScoreNote (see above, the OSC responder takes for granted that note messages have the format suitable for the function MakeScoreNote). Then, the constraint solver MySearcher is called: the method next invokes the solver and returns the next solution — in this example the next counterpoint note. Arbitrary arguments can be handed to the scripts Args argument (see above) by handing them as arguments to the solver's next method. We use this mechanism to forward the freshly received note SimNote to the script. Finally, the solver's output is tested whether it is the default solution nil (i.e., whether a timeout or failure happened). If this is not the case, then the note output by the solver is transformed into an OSC bundle (using MakeOSCNote, see above) and send to Supercollider.
{MyDumpOSC setResponder('/note' proc {$ Timetag Msg}
%% optional, for debugging
%% {Browse input#Start#Msg}
%% OSC to Strasheela note
SimNote = {MakeScoreNote Timetag Msg}
%% call solver
NewNote = {MySearcher next($ inputScore:SimNote)}
in
if NewNote \= nil
then %% transform note to OSC and send it back
{MySendOSC send({MakeOSCNote NewNote})}
else {Browse 'no solution found'}
end
end)}
The rest of this example is defined in Supercollider. The cantus firmus of this example is generated algorithmically in Supercollider by a number of patterns. A Supercollider pattern serves the creation of a sequence of values, possibly of infinite length. Patterns are often used to describe various parameters of a note sequence. In this section, we will define a pattern for each of the following note parameters: note durations, pitches and amplitudes. The explanation of these patterns will be brief, please see the Supercollider documentation for more information on its patterns.
For the sake of simplicity, the first pattern is extremely simple: all note amplitudes have the MIDI velocity 64.
~amps = 64.asStream;
The pattern for the durations is slightly more complex. The pattern randomly selects from three different sequence patterns. These sequence patterns are chosen such that the resulting durations always express a triple meter. For example, in each of these sequence patterns all its values sum up to three. The note durations are measured in seconds, but the pattern values will later be multipled by some tempo factor (see below).
~durs = Prand([Pseq([2.0, 1.0]), Pseq([1.0, 1.0, 1.0]), 3.0], inf).asStream;
The pitch pattern performs a random walk through the pitches in the C major scale, expressed by MIDI key-numbers. The walk starts at the tonic 60 (the third element, using zero-based indexing), the maximum pitch interval is a fourth up or downwards, but small intervals are preferred.
~pitches = Pwalk(// pitches in C major from g to c over 1 1/2 octaves
[55, 57, 59, 60, 62, 64, 65, 67, 69, 71, 72],
// steps up to 3 in either direction, but no repetition,
// and weighted toward positive
Pwrand([-3, -2, -1, 1, 2, 3],
[0.02, 0.05, 0.4, 0.4, 0.1, 0.03].normalizeSum, inf),
1, // reverse direction at boundaries
3 // start at tonic
).asStream;
Finally, this section starts the note generation process in Supercollider, sends and receives notes from Strasheela, and sends all notes to Supercollider's sound generation program scsynth for playback.
The following code segment generates the cantus firmus by using the patterns defined before. It defines a routine ~myRoutine, whose body is an infinite loop. Each loop iteration fetches a value from each of the above patterns (using the method next), uses these values to construct a new note which is send as message to Strasheela and played back on the scsynth. The loop then waits for the duration of the note, before constructing and sending the next note.
Please note the two variables ~latency and ~tempoFactor. The tempo factor is simply a factor for the note durations, allowing to speed up or slow down the music. To allow for a synchronised playback of both voices, the latency compensates for the processing time needed by Strasheela. The latency is used as timetag when OSC bundles are send to Strasheela and the scsynth. The scsynth delays the playback of the received notes by this amount (50 msecs in this example). Strasheela preserves this timetag in its newly generated note (see above) so that exactly the same timetag is used when this note is send to the scsynth (see below).
~latency = 0.05; // in secs
~tempoFactor = 0.6;
~myRoutine = Routine.new({
inf.do({ arg i;
var dur, pitch, amp;
// fetch pattern values
dur = ~durs.next * ~tempoFactor;
amp = ~amps.next;
pitch = ~pitches.next;
// send note to Strasheela
~mySendOSC.sendBundle(~latency, ['/note', (dur*1000).asInt, pitch, amp]);
// play note on scserver
s.makeBundle(~latency,
{Synth("Strasheela-playback", [\dur, dur, \freq, pitch.midicps, \amp, amp/127, \ffreq, 500, \pan, 0.9]);});
dur.wait;
});
"done".postln;
});
The following function is called whenever a note is received from Strasheela (cf. the definition of a recvOSCfunc function above). This note is then played back on the scsynth.
thisProcess.recvOSCfunc = { arg time, addr, msg;
if ( addr.port != 57110, // ignore scsynth messages
{ var address, dur, pitch, amp;
# address, dur, pitch, amp = msg;
s.makeBundle(time-thisThread.seconds,
{Synth("Strasheela-playback", [\dur, dur, \freq, pitch.midicps, \amp, amp/127, \ffreq, 2000, \pan, -0.9]);});
};)
};
We are finally in the position to run the example. The method play starts the routine. The routine creates note events with the patterns, sends these notes to Strasheela and the scsynth. Strasheela creates a new notes, which it sends back. The recvOSCfunc receives these notes from Strasheela and sends them to the scsynth. Simultaneous notes are played exactly in sync (if the latency is high enough).
SystemClock.play(~myRoutine);
This example demonstrated how a realtime music CSP can be defined in Strasheela. Please note that the implementation is not optimised in any way. For example, the implementation is purely declarative (no stateful operation), generating continuously lots of data which has to be garbage collected. Nevertheless, given a high enough latency, the output of the example is perfectly timed.
Garbage collection is indeed performed repeatedly while the example is running, and unlike Supercollider, Oz has not a realtime garbage collector. In Oz, the program execution must be shortly interrupted for garbage collection. Nevertheless, its garbage collection algorithm (copying dual-space algorithm) is fast for programs which require little active memory size. Moreover, larger applications can be split into multiple processes (using Oz' support for distributed programming), where each process has its own local garbage collection. If the time-critical parts of an Oz application run in small processes, their local garbage collection will be fast.
In this example, the input voice is created automatically by Supercollider. Alternatively, you may hook a MIDI keyboard into Supercollider, transform the incoming MIDI notes into OSC messages, and send them to Strasheela. You can then play some cantus firmus in realtime, and Strasheela generates a second voice for it. Please note that for a synchronised output, you still need to delay the playback of all notes by some latency.
Supercollider and Strasheela communicate via OSC in this example. Similarly, you can use Straheela's realtime constraint programming facilities for any other music programming environment which supports OSC, for example Pure Data.
1. Note that this interface calls dumpOSC in a terminal (xterm), and sends its output to Oz via a socket with netcat. Starting dumpOSC in a terminal is necessary, because for unknown reasons dumpOSC refuses to output anything when called by Oz directly in a pipe (for details, see postings in the mailing lists osc_dev@create.ucsb.edu, and users@mozart-oz.org, on the 7 September 2007 and following days). This interface relies thus on the following applications, which must be installed: sendOSC, dumpOSC, xterm, and netcat (nc). On most Unixes, xterm is already there. On MacOS, however, the X11 application must be installed in order to make xterm available (please find this application on your MacOS install CDs).
Additionally, all these applications must be specified in the Strasheela environment (if they are not in the PATH). For example, add lines like the following to your Oz init file ~/.ozrc.
{Init.putStrasheelaEnv sendOSC "/path/to/sendOSC"}
The respective Strasheela environment variables are sendOSC (its default value is 'sendOSC'), dumpOSC (default 'dumpOSC'), xterm (default 'xterm'), netcat (default 'nc'), and 'X11.app' (default '/Applications/Utilities/X11.app'). The environment variable 'X11.app' is only required on MacOS.
The original dumpOSC delays the printout of bundles (when called in a pipe as this interface does) and it is recommended to apply the dumpOSC patch available here (or simply replace the original file dumpOSC.c with the already patched dumpOSC.c in the same directory before compiling dumpOSC).
2. By default, only a single element is available in the arguments Args.outputScores and Args.inputScores in order to save RAM (many previous solutions may accumulate otherwise, and they may be copied often during the search process). However, the maximum length of these arguments can be changed, see the RT documentation for details.
3. The RT.scoreSearcherWithTimeout argument distroArgs expects a record in the same format as solvers like SDistro.exploreOne and friends used in the examples before, see the SDistro documentation.