How to Build a Speech to Emotion Converter with the Web Speech API and Node.js

Have you ever wondered if it‘s possible to detect the emotional tone of spoken words using web technologies? With the advancements in speech recognition and sentiment analysis, we can now build applications that can understand not only what is being said but also the underlying emotions conveyed through speech.

In this blog post, we‘ll explore how to create a speech to emotion converter using the Web Speech API and Node.js. We‘ll leverage the power of the Web Speech API to capture and convert speech to text, and then utilize sentiment analysis techniques to determine the emotional tone of the spoken words. By the end of this post, you‘ll have a working prototype that can detect and display the emotions behind spoken sentences.

Prerequisites and Setup

Before we dive into the implementation details, let‘s ensure we have the necessary tools and libraries set up. Here‘s what you‘ll need:

• Node.js: Make sure you have Node.js installed on your system. You can download it from the official Node.js website (https://nodejs.org).

• Express.js: We‘ll use Express.js, a popular web application framework for Node.js, to set up our server. Install it by running the following command in your terminal:

npm install express

• Web Speech API: The Web Speech API is a powerful tool that allows web applications to incorporate speech recognition and synthesis capabilities. It is supported by modern web browsers such as Google Chrome, Firefox, and Safari. No additional installation is required.

• Sentiment Analysis Library: We‘ll use a sentiment analysis library to determine the emotional tone of the speech text. For this example, we‘ll use the "Sentiment" library. Install it by running the following command:

npm install sentiment

Now that we have the prerequisites in place, let‘s create the project directory and set up the necessary files:

1. Create a new directory for your project and navigate into it.

2. Initialize a new Node.js project by running the following command:

npm init -y

3. Create a new file named server.js in the project directory. This file will contain the server-side code.

4. Create a new directory named public in the project directory. This directory will hold the client-side files.

5. Inside the public directory, create an index.html file and a script.js file.

With the project structure set up, we‘re ready to start implementing the speech to emotion converter.

Implementing Speech Recognition

The Web Speech API provides a convenient way to capture speech input from the user‘s microphone and convert it to text. Let‘s see how we can utilize this API to implement speech recognition in our application.

Open the script.js file in the public directory and add the following code:

const recognition = new webkitSpeechRecognition();
recognition.continuous = true;
recognition.interimResults = true;
recognition.lang = ‘en-US‘;

recognition.onresult = function(event) {
  const transcript = event.results[event.results.length - 1][0].transcript;
  console.log(‘Speech recognized:‘, transcript);
  // Send the speech text to the server for emotion analysis
  analyzeSpeech(transcript);
};

recognition.onerror = function(event) {
  console.error(‘Speech recognition error:‘, event.error);
};

function startRecognition() {
  recognition.start();
}

function stopRecognition() {
  recognition.stop();
}

In this code snippet, we create a new instance of the webkitSpeechRecognition object, which provides the speech recognition functionality. We set continuous to true to enable continuous speech recognition, interimResults to true to receive interim results, and lang to ‘en-US‘ to specify the language as English (United States).

We define the onresult event handler to capture the recognized speech. Inside this handler, we extract the transcript from the event results and log it to the console. We also call the analyzeSpeech function, which will send the speech text to the server for emotion analysis.

The onerror event handler is defined to handle any errors that may occur during speech recognition.

Finally, we define the startRecognition and stopRecognition functions to control the start and stop of speech recognition.

Server-side Processing with Node.js

Now that we have the speech recognition set up on the client-side, let‘s move to the server-side and implement the emotion analysis using Node.js and the Sentiment library.

Open the server.js file in the project directory and add the following code:

const express = require(‘express‘);
const sentiment = require(‘sentiment‘);

const app = express();
const port = 3000;

app.use(express.static(‘public‘));

app.post(‘/analyze‘, express.json(), (req, res) => {
  const text = req.body.text;
  const result = sentiment.analyze(text);
  res.json(result);
});

app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}`);
});

In this code, we import the express and sentiment modules. We create an instance of the Express application and set the server port to 3000.

We use app.use(express.static(‘public‘)) to serve static files from the public directory, allowing the client-side files to be accessible.

We define a POST route /analyze that accepts JSON data in the request body. Inside this route handler, we extract the text property from the request body, which contains the speech text sent from the client-side.

We pass the speech text to the sentiment.analyze function provided by the Sentiment library. This function analyzes the emotional tone of the text and returns a result object containing the sentiment score and other relevant information.

Finally, we send the sentiment analysis result back to the client as a JSON response.

Emotion Analysis and Scoring

The Sentiment library uses a lexicon-based approach to determine the emotional tone of a given text. It assigns sentiment scores to individual words and phrases based on a predefined dictionary of sentiment-bearing terms.

The sentiment scores range from -5 (highly negative) to 5 (highly positive), with 0 indicating a neutral sentiment. The library calculates the overall sentiment score of the text by summing up the scores of the individual words and applying additional heuristics.

For our speech to emotion converter, we can map the sentiment scores to specific emotions based on predefined thresholds. Here‘s an example of how we can interpret the sentiment scores:

• Score >= 2: Positive emotion (e.g., happy, excited)
• Score >= 0 and < 2: Neutral emotion
• Score < 0: Negative emotion (e.g., sad, angry)

You can adjust these thresholds based on your specific requirements and the desired granularity of emotion detection.

Client-side Integration

With the server-side emotion analysis in place, let‘s integrate it with the client-side to display the detected emotions to the user.

Open the script.js file in the public directory and add the following code:

function analyzeSpeech(text) {
  fetch(‘/analyze‘, {
    method: ‘POST‘,
    headers: {
      ‘Content-Type‘: ‘application/json‘
    },
    body: JSON.stringify({ text: text })
  })
    .then(response => response.json())
    .then(result => {
      displayEmotion(result);
    })
    .catch(error => {
      console.error(‘Error analyzing speech:‘, error);
    });
}

function displayEmotion(result) {
  const emotionElement = document.getElementById(‘emotion‘);
  const score = result.score;

  if (score >= 2) {
    emotionElement.textContent = ‘😊 Positive‘;
  } else if (score >= 0 && score < 2) {
    emotionElement.textContent = ‘😐 Neutral‘;
  } else {
    emotionElement.textContent = ‘😢 Negative‘;
  }
}

In the analyzeSpeech function, we send a POST request to the /analyze endpoint on the server, passing the speech text in the request body as JSON.

Once the server responds with the sentiment analysis result, we call the displayEmotion function, passing the result object.

Inside the displayEmotion function, we extract the sentiment score from the result object and use conditional statements to determine the corresponding emotion based on the predefined thresholds.

We then update the textContent of the emotionElement (assuming you have an HTML element with the ID "emotion") to display the detected emotion along with an appropriate emoji.

Putting It All Together

Finally, let‘s bring everything together by updating the index.html file to include the necessary HTML elements and buttons to control speech recognition and display the detected emotions.

Open the index.html file in the public directory and add the following code:

<!DOCTYPE html>
<html>
<head>
  <title>Speech to Emotion Converter</title>
</head>
<body>

  <button onclick="startRecognition()">Start</button>
  <button onclick="stopRecognition()">Stop</button>
  <div id="emotion"></div>

  <script src="script.js"></script>
</body>
</html>

In this HTML code, we include two buttons: "Start" and "Stop". The "Start" button triggers the startRecognition function to begin speech recognition, while the "Stop" button triggers the stopRecognition function to stop speech recognition.

We also have a <div> element with the ID "emotion" where the detected emotion will be displayed.

Finally, we include the script.js file at the end of the <body> section to ensure that the HTML elements are loaded before the JavaScript code is executed.

Running the Application

To run the speech to emotion converter application, follow these steps:

1. Open a terminal and navigate to the project directory.

2. Run the following command to start the Node.js server:

node server.js

3. Open a web browser and visit http://localhost:3000.

4. Click the "Start" button to begin speech recognition.

5. Speak into the microphone, and the application will capture your speech, convert it to text, analyze the emotional tone, and display the detected emotion on the webpage.

6. Click the "Stop" button to stop speech recognition when you‘re done.

Conclusion

Congratulations! You have successfully built a speech to emotion converter using the Web Speech API and Node.js. This application demonstrates how we can leverage speech recognition and sentiment analysis technologies to detect the emotional tone of spoken words.

Throughout this blog post, we covered the key steps involved in creating the speech to emotion converter, including setting up the project, implementing speech recognition using the Web Speech API, performing server-side emotion analysis with Node.js and the Sentiment library, and integrating the client-side and server-side components.

The speech to emotion converter has numerous potential applications, such as sentiment analysis in customer support, emotion detection in virtual assistants, and even in personal well-being and mental health monitoring.

Remember that this is just a starting point, and there are many opportunities for further enhancements and optimizations. You can explore techniques like fine-tuning the sentiment analysis model, supporting multiple languages, and incorporating more advanced machine learning algorithms for improved emotion detection accuracy.

I encourage you to experiment with the code, customize it to suit your specific needs, and build upon this foundation to create even more exciting and innovative applications.

If you have any questions or feedback, please feel free to reach out. Happy coding, and may your applications be emotionally intelligent!