3-5 Sound

p5.js Libraries

We have been focusing on graphics, images, and fonts because these are core features built-in to p5.js. It’s also possible to extend these basic capabilities by using a variety of free 3^rd party libraries. A library is a collection of related functions, variables, and objects that extend the functionality of a programming language. Some of the commonly used libraries are automatically accessible in the p5.js online editor, but some require us to upload a .js library file and make some (minor) configuration changes.

If you look at the Libraries section of the p5.js online reference [ https://p5js.org/libraries/ ] you’ll find information on dozens of p5.js libraries that you can use to do interesting things with p5.js. Today we’ll look at libraries related to adding sound to our programs.

p5.sound Library

The p5.sound library allows us to load and play files of a variety of sound formats including .mp3, .wav, and .ogg. Although this is a 3^rd party library, the p5.js online editor automatically makes this library available to us in our programs.

As with images, we will define a preload() function to load these external media before setup(). It’s also important to make sure that the file extension matches the format of the sound file.

- .mp3 files are optimized for playing longer, complete songs (not short sound effects)
- .wav files are best for short, quick sound effects (not music)
- .ogg files provide the best of both worlds and work well for any kind of music or sound effect, in my experience.

Using free software like Audacity [ https://www.audacityteam.org/ ] makes it easy to convert different audio file formats and create/edit your own sound effects or music.

Here’s an example that will start/stop music whenever the mouse is clicked on the canvas.

let myMusic

function preload() {
  myMusic = loadSound("sound/music.ogg")
}

function setup() {
  createCanvas(400, 400)
  myMusic.setVolume(0.5) // 50% Volume
  myMusic.rate(0.5)      // 50% Speed
  myMusic.reverseBuffer()  // Reverse
  myMusic.play()
}

function draw() {
  background(220)
  fill(255,0,0)
  textAlign(CENTER)
  text("Click mouse to stop/restart music!", width/2, height/2)   }

function mousePressed() {
  if (myMusic.isPlaying()) {
    myMusic.stop()
    //myMusic.pause()
  } 
  else {
    myMusic.play()
  } 
}

let myMusic

function preload() {

myMusic = loadSound("sound/music.ogg")

}

function setup() {

createCanvas(400, 400)

myMusic.setVolume(0.5) // 50% Volume

myMusic.rate(0.5) // 50% Speed

myMusic.reverseBuffer() // Reverse

myMusic.play()

}

function draw() {

background(220)

fill(255,0,0)

textAlign(CENTER)

text("Click mouse to stop/restart music!", width/2, height/2) }

function mousePressed() {

if (myMusic.isPlaying()) {

myMusic.stop()

//myMusic.pause()

}

else {

myMusic.play()

}

The preload() function works much like with loading images, except we are loading a sound file. In this case, the music is in .ogg format, and I’ve uploaded it into a folder called sound. The loadSound() function actually instantiates an object, and that object includes a variety of methods to control the sound.

In the setup() function, I use the setVolume() method to control the volume level for the music. You can use a value between 0.0 and 1.0 here corresponding to the decimal percentage you want; for example, use 0.3 for a volume level of 30%. The play() method starts the music. If you want the music to play over and over again you can call the loop() method instead of play().

In the mousePressed() function I’m using a method called isPlaying() to determine the status of the music. If the music is currently playing, then I use the stop() method to stop it, or if it’s not playing then I call the play() method to start it. If you try this example you’ll note that the music starts over from the beginning each time. To have the music continue from where it was last stopped, use the pause() method instead of stop() on line 23:

myMusic.pause()

There are dozens of other more advanced methods in the p5.sound library if you are interested to learn more. For fun, check out the rate() method to control the speed of playback, and the reverseBuffer() method to play sound backwards. Listen to some of your favourite songs to see if they have hidden messages in them!

p5.speech Library

The p5.speech library can be used to convert text (i.e., a “string”) into speech! Unlike the p5.sound library, we need to do 2 extra steps to be able to use this library.

First, you need to download the p5.speech library [ https://idmnyu.github.io/p5.js-speech/#download ] – I recommend you save it to your desktop so it’s easy to find. Once you have downloaded the p5.speech.js file, upload it to the same location as your sketch.js file in the p5.js online editor.

Second, we need to add a special line of code to the index.html file in the p5.js online editor to get access to this library. This is what the index.html file should contain.

<!DOCTYPE html>
<html lang="en">
  <head>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.3.1/p5.js"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.10.2/addons/p5.sound.min.js"></script>
    <script src="p5.speech.js"></script>
    <link rel="stylesheet" type="text/css" href="style.css">
    <meta charset="utf-8" />

  </head>
  <body>
    <script src="sketch.js"></script>
  </body>
</html>

<!DOCTYPE html>

<head>

</head>

<body>

</body>

</html>

You need to add line 6 (only) to the index.html file so that p5.js can load the p5.speech.js source file from the same location as our other files in the p5.js online editor.

Now, here’s an example demonstrating some of the text-to-speech capabilities:

let myVoice

function setup() {
  createCanvas(400, 400)
  myVoice = new p5.Speech()
}

function draw() {
  background(220)

  fill(255,0,0)
  textAlign(CENTER)
  text("Click to hear a message.", width/2, height/2)   }

function mousePressed() {
  myVoice.listVoices()
  myVoice.setVoice("Victoria")
  myVoice.speak('p5.js is awesome!')
}

let myVoice

function setup() {

createCanvas(400, 400)

myVoice = new p5.Speech()

}

function draw() {

background(220)

fill(255,0,0)

textAlign(CENTER)

text("Click to hear a message.", width/2, height/2) }

function mousePressed() {

myVoice.listVoices()

myVoice.setVoice("Victoria")

myVoice.speak('p5.js is awesome!')

}

On line 5 we instantiate a Speech object. When the mouse is clicked, the listVoices() method will display a list of available voices in the console area. You can think of this list of voices as like a list of fonts. Line 17, uses the setVoice() method to select “Victoria” as the voice. Finally, the speak() method will cause whatever “string” is given to be output as speech on your speakers. Pretty cool, but wait… there’s more!

Speech Recognition

Wouldn’t it be nice if the p5.speech library could provide voice recognition too? It can! To do this we instantiate a SpeechRec object (line 5, below) which provides instance variables and methods to handle speech recognition. Here’s a simple example:

let myRec

function setup() {
  createCanvas(400, 400)
  myRec = new p5.SpeechRec()
  myRec.onResult = showResult
  myRec.continuous = true
  myRec.start()
}

function draw() {
  background(220)

  fill(255,0,0)
  textAlign(CENTER)
  text("Say something!", width/2, height/2)   
}

function showResult() {
  print("You said: " + myRec.resultString)
}

let myRec

function setup() {

createCanvas(400, 400)

myRec = new p5.SpeechRec()

myRec.onResult = showResult

myRec.continuous = true

myRec.start()

}

function draw() {

background(220)

fill(255,0,0)

textAlign(CENTER)

text("Say something!", width/2, height/2)

}

function showResult() {

print("You said: " + myRec.resultString)

}

Note: When you run this example, your web browser will ask you to grant access to the microphone. You will need to allow access for this code to work.

Let’s start at the bottom with the showResult() function – you can name this function anything you like. In showResult() I’m simply displaying the line of text (i.e., in the myRec.resultString instance variable) that was recognized from the users’ microphone to the console area.

On line 6, in the setup() function, you’ll see that I assign the name of the showResult() function (without parentheses) to the myRec.onResult instance variable. This tells p5.js that when microphone input has been recognized that the function showResult() should be called automatically. Line 7 ensures that the program will handle as many speech inputs as the user gives and that the showResult() function will be called each time. Finally, line 8 starts the speech recognition process. For such a small amount of code, this is a very powerful program!

You Try!

1. Create a music player with buttons to start, stop, pause, increase volume, and decrease volume.
2. Write a program that changes the background colour of the canvas to red, green, or blue based on voice commands.
3. Create a hands-free drawing program with commands: up, down, left, right, stop, red, green, and blue. For an extra challenge, recognize the numbers 1 through 9 to adjust the line thickness.