In this tutorial, we will guide you on how to install Flume on Elementary OS. Flume is a distributed service for collecting, aggregating, and moving huge amounts of log data from different sources into a centralized store. Flume supports various sources like HTTP, JMS, Twitter, and many more.
Before we begin, you need to ensure that you have the following:
The first step is to ensure that your system has Java installed, as Flume requires it to run. To install Java, run the following command in the terminal:
sudo apt update
sudo apt install default-jdk
Next, we need to download Flume from the Apache website. To download Flume, you can use either of the following commands:
wget https://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
or
curl -O https://www.apache.org/dist/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
Once the download is completed, we need to extract the Flume archive. To do so, navigate to the directory where you have downloaded Flume and run the following command:
tar -xvzf apache-flume-1.9.0-bin.tar.gz
This will extract the files to a directory called 'apache-flume-1.9.0-bin'.
Before running Flume, we need to create a configuration file. Flume uses an Apache Avro protocol to communicate between the source and the sink. So, we will create a configuration file ‘flume.conf’ in the 'apache-flume-1.9.0-bin' directory for the Avro source.
Run the following command to create a new configuration file:
nano apache-flume-1.9.0-bin/flume.conf
And add the following lines to the file:
a1.sources = avroSrc
a1.channels = memoryChannel
a1.sinks = hdfsSink
a1.sources.avroSrc.type = avro
a1.sources.avroSrc.bind = localhost
a1.sources.avroSrc.port = 41414
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.hdfs.path = /flume/events
a1.sinks.hdfsSink.hdfs.filePrefix = events-
a1.sinks.hdfsSink.hdfs.fileSuffix = .log
a1.sinks.hdfsSink.hdfs.fileType = DataStream
a1.sinks.hdfsSink.hdfs.writeFormat = Text
a1.sinks.hdfsSink.channel = memoryChannel
a1.channels.memoryChannel.type = memory
a1.channels.memoryChannel.capacity = 1000
a1.channels.memoryChannel.transactionCapacity = 100
# Bindings
a1.sources.avroSrc.channels = memoryChannel
a1.sinks.hdfsSink.channel = memoryChannel
Save the changes by pressing Ctrl + X, then Y, and then Enter.
Finally, we are ready to run Flume. To run Flume, navigate to the 'apache-flume-1.9.0-bin' directory and run the following command:
./bin/flume-ng agent --conf ./conf --conf-file ./flume.conf --name a1 -Dflume.root.logger=INFO,console
This command will start Flume with the configuration we created (flume.conf), and logs will be shown on the console (INFO level).
In this tutorial, we have successfully installed and configured Flume on Elementary OS. Flume is now ready to collect, aggregate, and move huge amounts of log data from various sources into a centralized store.
If you want to self-host in an easy, hands free way, need an external IP address, or simply want your data in your own hands, give IPv6.rs a try!
Alternatively, for the best virtual desktop, try Shells!
