Building an Event-driven Architecture using AWS Lambda & DataChannel

In March this year, after introducing data orchestration with DataChannel we added two new data analytics nodes: ‘Tableau + Power BI’. Now, we’re excited to announce another powerful addition: the ‘Lambda Function’ node. This new feature allows you to trigger any Lambda function in your directory directly from the DataChannel console, giving you control to automate your data flow and manage events within your Amazon S3 buckets.

If you're new to data orchestration and want to learn more about Lambda functions and how they gel up with DataChannel, this blog is for you. By the end of this blog, you’ll have a clear understanding and who knows, you might even start using DataChannel for your own data orchestration needs. Let’s begin with a quick introduction to AWS Lambda, serverless computing, why serverless matters, and some common use cases.

What is an AWS Lambda Function?

Lambda functions are a way of automatically triggering a function in response to an event. That event could be a new data object or point addition to an S3 bucket. Lambda functions are serverless, meaning you don't have to worry about computing power, automatic scaling, server maintenance and infrastructure needs, detailed logging, or system capacity provisioning—AWS handles it all through Lambda.

Why Serverless Computing?

Serverless Computing is an auto-scaling technology that allows developers to use backend services without worrying about managing some on-premises servers. With serverless computing, you pay only for the services you use, at no additional charge for increased demand or activity. A similarity that could help you better understand serverless computing is switching from a mobile phone data plan with fixed monthly limits to one that only calculates each data byte.

Serverless & FaaS (Function-as-a-Service)

Although FaaS and servers are roughly the same, serverless consists of cloud providers maintaining all backend services and infrastructure-related requirements for running applications (code).

FaaS (Function as a Service) is a platform that provides serverless architecture, orchestration, AWS lambdas, Google, Azure, and more. A serverless function is an application code that performs a specific action.

Serverless Platform: AWS Lambda 

AWS Lambda is an event-controlled serverless computing platform offered by Amazon as part of Amazon Web Services (AWS). This is a computer service that runs code as an answer to events and automatically manages the computing resources required by this code. It works again with serverless technology and doesn't need to create series-related requirements. Simply place your code in the lambda and run it.

How does AWS Lambda work?

The code you want Lambda to run is called a Lambda Function. The function responds to the “Events” such as object uploads to Amazon S3 bucket, updates to dynamo DB table, in-app purchases, etc. For example, these “Events” can be ‘Create Events’ like PUT, POST, COPY objects from or to S3 Bucket.

You can create a Lambda function by uploading your code as the function or by writing one on your own within the AWS Console.

AWS Lambda Components

The main purpose of AWS Lambda is to create applications based on events that have the opportunity to be activated by various AWS events.

If a situation causes several concurrent events, AWS Lambda will trigger several copies of the function, and Lambda will become the function (FAAS) of the service.

The three main components are:

Function: The actual code is here to run the task. Lambda currently supports more than 15 conditions, with new additions scheduled to occur a little later in 2024.

Configuration: This component gives information about the function and its execution.

Event Source: The event source is not one of the main components of the lambda function. However, it specifies the event that will trigger the lambda function run, and it belongs to any supported AWS or third-party service.

Log streams: This component helps users identify logs for different lambda function runs; these log streams can be accessed either using Cloudwatch directly or via fetching custom logs manually.

AWS Lambda Use Cases

A simple use case could be resizing an image present in your Amazon S3 bucket. You would first need to upload the image in your S3 bucket (using the lambda function), followed by code execution to resize your image to make it compatible with different storage devices. 

The possible steps execution sequence can be:

✓ Once the upload is successful, an event notification is triggered.
✓ This notification invokes a pre-configured AWS Lambda function.
✓ The Lambda function processes the image and resizes it.
✓ After resizing, the modified images are saved back into the S3 bucket.

Event-driven ETL Architecture

Let’s say a website’s order data is stored directly in an operational database: Amazon DynamoDB. Using Lambda functions (which will run in response to a new order-related transactional entry) this data can be transformed into a structured format suitable for the data warehouse: Amazon Redshift from which the data can be further moved into any BI tool in the form of tables or views. 

What’s Next?

In our next blog, we’ll talk about how we have built our integration with the Lambda function and how it enables our users to automate event-driven tasks via DataChannel. We’ll deep dive into the use cases that can be covered using this integration while leveraging the same functionalities as lambda’s management console.