Building a Community Sign-up App with Serverless, StepFunctions, and StackStorm Exchange – Episode 3

Welcome back, serverless enthusiasts! We‘re now in the third installment of our series on building a real-world serverless application on AWS. If you missed the previous episodes, I highly recommend starting with Episode One and Episode Two to get the full context.

In this episode, we‘ll take a deep dive into AWS StepFunctions and how they enable us to orchestrate our Lambda functions into a robust serverless workflow. As a full-stack developer who has built my fair share of complex distributed systems, I‘ve come to appreciate the power and flexibility of serverless workflows, especially for coordinating loosely coupled microservices. StepFunctions have become an indispensable tool in my serverless toolkit, and I‘m excited to share my experience with you.

The Rise of Serverless Workflows

Before we jump into the technical details, let‘s take a step back and consider the broader context of serverless architectures and workflows.

Serverless computing has fundamentally changed the way we build and operate applications in the cloud. By abstracting away the underlying infrastructure, serverless platforms like AWS Lambda enable developers to focus solely on writing business logic in the form of functions. These functions are event-driven, stateless, and ephemeral, providing a highly scalable and cost-effective model for executing code.

However, as serverless applications grow in complexity, composing and coordinating multiple functions becomes increasingly challenging. Traditional approaches like direct function-to-function invocations or managing state in a shared database can quickly lead to tight coupling, fragile integrations, and operational complexity.

This is where serverless workflows come into play. Workflows provide a declarative and visual way to orchestrate serverless functions, manage their execution state, handle errors and retries, and exchange data between them.

AWS StepFunctions, launched in 2016, is a fully managed service that makes it easy to build and run serverless workflows. StepFunctions have since become a key component of the serverless ecosystem, used by countless organizations to build everything from data processing pipelines and ETL jobs to order fulfillment systems and machine learning workflows.

To illustrate the advantages of serverless workflows, let‘s consider a simple example. Suppose we have three Lambda functions:

1. ProcessPayment: Validates and processes a customer‘s payment.
2. UpdateDatabase: Persists the order details to a database.
3. SendConfirmation: Sends an order confirmation email to the customer.

In a traditional approach, the functions might be invoked directly by an API Gateway, like this:

While this works for simple use cases, it has several drawbacks:

• The API Gateway needs to know about each function and their invocation order.
• The functions are tightly coupled, making it harder to change or reuse them independently.
• There‘s no built-in mechanism for handling errors or retrying failed executions.
• The execution state is lost between function invocations, making it difficult to track and debug.

Now, let‘s see how the same use case could be implemented as a serverless workflow with StepFunctions:

Here, the workflow acts as a central orchestrator, invoking the functions in the desired order and managing the execution state. The benefits are clear:

• The workflow is decoupled from the individual functions, allowing them to evolve independently.
• The visual representation makes it easy to understand and reason about the overall flow.
• Built-in error handling and retry mechanisms improve resilience.
• The execution state is automatically captured, enabling end-to-end visibility and debugging.

With this background in mind, let‘s see how we can apply serverless workflows to our community sign-up application using AWS StepFunctions.

Defining Serverless Workflows with Amazon States Language

At the core of AWS StepFunctions is the Amazon States Language (ASL), a JSON-based structured language for defining the steps and transitions in a workflow. ASL provides a wide range of constructs for modeling different workflow patterns, including:

• Task states for executing a single Lambda function or activity.
• Choice states for branching the flow based on conditions.
• Wait states for introducing delays or timeouts.
• Parallel states for executing multiple branches concurrently.
• Map states for dynamically iterating over a list of items.

The basic structure of an ASL definition consists of a collection of states, each representing a step in the workflow. States are connected by transitions that specify the next state to move to after the current state completes.

Here‘s a simplified version of our sign-up workflow definition:

{
  "Comment": "Community Sign-up Workflow",
  "StartAt": "RecordSignUp",
  "States": {
    "RecordSignUp": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:record-signup",
      "Next": "SendWelcomeEmail"
    },
    "SendWelcomeEmail": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:send-welcome-email",
      "Next": "UpdateDataWarehouse"
    },
    "UpdateDataWarehouse": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:update-data-warehouse",
      "End": true
    }
  }
}

In this example:

• The workflow starts at the RecordSignUp state, which invokes a Lambda function to persist the user‘s sign-up information.
• After recording the sign-up, it transitions to the SendWelcomeEmail state to send a welcome email to the user.
• Finally, the UpdateDataWarehouse state updates a downstream data warehouse with the new user‘s information and ends the workflow.

Of course, real-world workflows are often more complex, involving multiple paths, error handling, and data transformations between states. ASL provides a rich set of features for modeling such advanced workflows.

One particularly powerful feature is the ability to pass data between states using the ResultPath, OutputPath, and Parameters fields. These allow you to selectively include or exclude data from the input, manipulate it using built-in intrinsic functions, and pass it to subsequent states.

Here‘s a more realistic example that demonstrates data flow between states:

{
  "StartAt": "GetUserProfile",
  "States": {
    "GetUserProfile": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:get-user-profile",
      "ResultPath": "$.profile",
      "Next": "UpdateDatabase"
    },
    "UpdateDatabase": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:update-database",
      "Parameters": {
        "username.$": "$.profile.username",
        "email.$": "$.profile.email"  
      },
      "ResultPath": null,
      "Next": "SendEmail"
    },  
    "SendEmail": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-west-2:123456789012:function:send-email",
      "Parameters": {
        "recipient.$": "$.profile.email",
        "subject": "Your profile has been updated"
      },
      "End": true
    }
  }
}

Here‘s what‘s happening:

1. The GetUserProfile state invokes a Lambda function to retrieve the user‘s profile information, and stores the result under the $.profile key in the state‘s output using ResultPath.

2. The UpdateDatabase state invokes a Lambda function to update the user‘s profile in a database. The Parameters field is used to map specific properties from the input (e.g., $.profile.username) to the function‘s input. ResultPath is set to null to prevent the function‘s result from being merged with the input.

3. The SendEmail state invokes a Lambda function to send an email notification to the user. Again, Parameters is used to pass the recipient‘s email address and a static subject line to the function.

By carefully managing the flow of data between states, you can ensure that each function receives exactly the input it needs, while maintaining a clear separation of concerns.

Integrating StepFunctions with the Serverless Framework

Now that we have a basic understanding of ASL and how to define workflows, let‘s see how we can integrate StepFunctions into our serverless application using the Serverless Framework.

The serverless-step-functions plugin makes it easy to define and deploy StepFunctions alongside your Lambda functions and other AWS resources. Here‘s how to set it up:

1. Install the plugin:

   npm install --save-dev serverless-step-functions

2. Add the plugin to your serverless.yml:

   plugins:
     - serverless-step-functions

3. Define your StepFunctions in the stepFunctions section of serverless.yml:

   stepFunctions:
     stateMachines:
       signUpWorkflow:
         name: signup-workflow
         definition:
           StartAt: RecordSignUp
           States:
             RecordSignUp:
               Type: Task
               Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-recordSignUp
               Next: SendWelcomeEmail
             SendWelcomeEmail:
               Type: Task
               Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}:function:${self:service}-${opt:stage}-sendWelcomeEmail
               End: true

   Note how the plugin uses the #{AWS::Region} and #{AWS::AccountId} syntax to reference CloudFormation pseudo parameters, and the ${self:service} and ${opt:stage} syntax to reference Serverless Framework variables. This makes the workflow definition more reusable across different stages and regions.

4. Deploy your service:

   serverless deploy

   The plugin will package your StepFunction definition, create the necessary IAM roles, and deploy the workflow alongside your Lambda functions.

With just a few lines of configuration, we‘ve integrated a serverless workflow into our application!

Real-world StepFunctions Use Cases and Best Practices

StepFunctions are incredibly versatile and can be used to model a wide variety of workflows across different domains. Here are a few real-world use cases:

• E-commerce order processing: A workflow that orchestrates the various steps involved in processing an online order, from payment authorization and fraud detection to inventory updates and shipping label generation.

• Data processing pipelines: A workflow that coordinates a series of data processing tasks, such as extracting data from multiple sources, transforming it, and loading it into a data warehouse or data lake.

• Machine learning workflows: A workflow that manages the end-to-end lifecycle of a machine learning model, from data preparation and feature engineering to model training, evaluation, and deployment.

• Business process automation: A workflow that automates repetitive business processes, such as employee onboarding, expense approvals, or customer support ticketing.

Here are some best practices to keep in mind when building serverless workflows with StepFunctions:

• Use modular and reusable workflow definitions: Break down complex workflows into smaller, reusable sub-workflows that can be composed together. This makes your workflows more maintainable and easier to test.

• Manage state transitions carefully: Use ResultPath, OutputPath, and Parameters to control how data flows between states. Be mindful of passing only the necessary data to each state to avoid hitting size limits.

• Handle errors gracefully: Use Catch and Retry fields to define error handling and retry logic for different types of exceptions. This makes your workflows more resilient to transient failures.

• Monitor and optimize for cost: StepFunctions pricing is based on the number of state transitions and the total execution time. Monitor your workflows using CloudWatch metrics and set up alarms to detect anomalies. Optimize your workflows to minimize the number of transitions and the duration of each state.

• Test thoroughly: Test your workflows locally using tools like StepFunctions Local before deploying to production. Use a combination of unit tests, integration tests, and end-to-end tests to ensure the correctness of your workflows under different scenarios.

Conclusion

In this deep dive, we explored the world of serverless workflows and how AWS StepFunctions enable us to orchestrate Lambda functions into robust, scalable, and maintainable applications. We learned about the Amazon States Language and its key concepts, and saw how to integrate StepFunctions into a Serverless Framework project using the serverless-step-functions plugin.

As a seasoned developer who has built complex distributed systems using both traditional and serverless architectures, I can confidently say that serverless workflows are a game-changer. They provide a level of abstraction, flexibility, and operational simplicity that was previously unattainable.

Of course, serverless workflows are not a silver bullet and come with their own set of challenges and limitations. But when used judiciously and in combination with other serverless services, they can significantly accelerate development velocity and reduce operational overhead.

I encourage you to experiment with StepFunctions in your own projects and share your experiences with the community. Together, we can push the boundaries of what‘s possible with serverless architectures and build applications that are more resilient, scalable, and maintainable.

Until next time, happy coding!

References

• AWS StepFunctions Developer Guide
• Amazon States Language Specification
• Serverless Framework StepFunctions Plugin
• AWS StepFunctions Best Practices

"The future of computing is serverless, and AWS StepFunctions is at the forefront of this revolution."

— Dr. Werner Vogels, CTO, Amazon.com