If you haven’t started implementing analytics in your application yet, you’re behind the game. The most valuable asset in the modern era is Data, and this will grow in the future as well. So what is Analytics? Through Analytics, you can get feedback from your clients without asking them: you can collect data and see how your users are navigating, performing, visiting, etc… After collecting this data, you can come up with an analysis, target more useful features, and update your UI, hence making your application more user-friendly with your analyzed data. Basically, after you implement analytics, your application will become more mature, more convenient for users, and more successful for your business. In this Blog we will create an AWS Kinesis Data Firehose that delivers stream to S3 bucket (using AWS CDK with typescript), and put our records from our React app using Amplify. Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. Let us split the content into two parts: First, the Infrastructure. Then, the Frontend.
We are going to use CDK to provision our resources. It helps us to write our code using languages like typescript. I will focus on how to create the S3 bucket & Firehose, and how to give permission to our Unauthenticated users to use it. Although I highly suggest to dive deeply into how to create Cognito as well, our scope isn’t about authentication, so I won’t be focusing on that. You can check the official CDK documentation via this link .
Resource Diagram:
CDK Stack:
const bucket = new s3.Bucket(this, bucketName, {
bucketName,
});
const result = bucket.addToResourcePolicy(new iam.PolicyStatement({
actions: ['s3:GetObject'],
resources: [bucket.arnForObjects('file.txt')],
principals: [new iam.AccountRootPrincipal()],
}));
const s3Destination = new destinations.S3Bucket(bucket, {
dataOutputPrefix,
errorOutputPrefix,
bufferingInterval: Duration.minutes(bufferingInterval),
bufferingSize: Size.mebibytes(bufferingSize),
});
const stream = new firehose.DeliveryStream(this, 'Delivery Stream', {
deliveryStreamName: streamName,
destinations: [s3Destination],
});
bucketName: 'your-bucket-name',
streamName: 'your-stream-name',
dataOutputPrefix: 'myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}',
errorOutputPrefix: 'myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}',
bufferingInterval: 2,
bufferingSize: 8,
First we create our bucket, then we add the resource policy.
Our destination has the following parameters:
dataOutputPrefix for files successfully delivered to S3
errorOutputPrefix for failed records before writing them to S3
By default, the buffer size (bufferingSize) is 5 MiB and the buffer interval is 5
minutes (bufferingInterval) . But in our example I changed them to 2 minutes and
8 MiB
Incoming data is buffered before it is delivered to the specified destination. The
delivery stream will wait until the amount of incoming data has exceeded some
threshold
(the “buffer size”) or until the time since the last data delivery occurred exceeds
some
threshold (the “buffer interval”), whichever happens first.
Now, regarding our Cognito and how to be able to put records from frontend, we will add this policy to our Authenticated & Unauthenticated users. This is flexible, so you can choose which users to be able to send records to. In our case, I will add for both.
{
effect: iam.Effect.ALLOW,
actions: [
'firehose:PutRecord',
'firehose:PutRecordBatch'
],
resources: ['your-firehose-arn']
}
Amplify is a set of purpose-built tools and services that makes it quick and easy to use
our AWS resources, I will be using Analytics part here, for official documentation you
can visit this link
First, let us see how we can configure our Analytics
App.js
import { Analytics, AWSKinesisFirehoseProvider } from 'aws-amplify';
Amplify.configure(awsConfig);
Analytics.addPluggable(new AWSKinesisFirehoseProvider);
awsConfig is a JSON file that holds our configuration. Inside it, add this object
Analytics: {
AWSKinesisFirehose: {
region: REGION
}
}
Now let's add our fancy button
const onClick = async () => {
const now = new Date;
const data = {
id: now.getTime(),
action: 'Add Button',
component: 'Button',
user: 'the username of user',
source: 'Web',
};
try {
await Analytics.record({
data: data,
streamName: FIREHOSE,
}, 'AWSKinesisFirehose');
} catch (error) {
console.log(error);
}
}
<button
onClick={onClick}
>Add Button</button>
FIREHOSE is our Kinesis firehose name that we created previously from the CDK. Analytics.record is where we put our record to the Firehose. After 2 minutes we can view it from our S3 bucket. Our data is simple: I added some attributes just to show how it can scale, and you can send very useful data for Analytics or even do ML on it later
Finally our result inside our S3 bucket (the file should be downloaded)
{"id":1637695583710,"action":"Add Button","component":"Button","user":"the username of user","source":"Web"}
In this example, I tried to limit the content scope to our topic. I used a simple button component, but this can go beyond your expectations and creativity. You can collect very useful and beneficial data from your users, which would assist in enhancing and improving your applications!
