How to send data to Kinesis Firehose
You can send data to Firehose delivery stream directly or through other collection systems. Let’s take a look at a few examples.
Amazon Kinesis Agent
You can write to Amazon Kinesis Firehose using Amazon Kinesis Agent. Amazon Kinesis Agent is a stand-alone Java software application that offers a way to collect and send data to Firehose. The agent continuously monitors a set of files and sends new data to your Firehose delivery stream. You can install the agent on Linux-based server environments such as web servers, log servers, and database servers.
You could analyze a live stream of Apache access log data and aggregate the total request for each HTTP response type every minute.
Writing to a Firehose Delivery Stream using the AWS SDK. You can use the Firehose API to send data to a Firehose delivery stream using the AWS SDK for Java, .NET, Node.js, Python, or Ruby.
You could move from your internal queue to a Kinesis Firehose service.
Here‘s an example: Streaming Real-time Data into an S3 Data Lake at MeetMe
“While RabbitMQ consumers are implemented elsewhere (such as Flume), we opted to develop our own versions to enable integration with Firehose API”
Using AWS Lambda function to forward Stream data to Kinesis Firehose. It’s possible to forward data from Amazon Kinesis Streams or Amazon DynamoDB to Amazon Kinesis Firehose.
You could create a Lambda function that reads the streams data and writes to the Amazon Kinesis Firehose delivery stream
Fluentd output plugin permits to send events to Amazon Kinesis Firehose (via API).
You could use the Fluentd’s Amazon Kinesis Output plugin to aggregate semi-structured logs in real-time.
AWS Service Proxy
Using AWS API Gateway via Service proxy. It’s possible to create an API Gateway as a proxy to an AWS service, such as Amazon SNS, Amazon S3, Amazon Kinesis.
One use case where it’s valuable to have immediate insights is analyzing clickstream data. In the world of digital advertising, an impression is when an ad is displayed in a browser and a clickthrough represents a user clicking on that ad. All client web requests can be handled by Amazon API Gateway. This handles the web request traffic to get data into your analytics pipeline by being a service proxy to an Amazon Kinesis.