Flume install and config

Flume install and config

1) Flume Install:

mkdir /usr/lib/flume

chmod 777 /usr/lib/flume

cd /usr/lib/flume

wget http://mirror.olnevhost.net/pub/apache/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz

tar -xvf apache-flume-1.6.0-bin.tar.gz

cd apache-flume-1.6.0-bin


2) Update global environment variable:

Set the following environment variable in /etc/bash.bashrc at the end of the file.

This will make the flume configuration available for all users of the system.

export FLUME_CONF_DIR=/usr/lib/flume/apache-flume-1.6.0-bin/conf
export FLUME_HOME=/usr/lib/flume/apache-flume-1.6.0-bin
export FLUME_CLASS_PATH=/usr/lib/flume/apache-flume-1.6.0-bin/lib
export PATH="$FLUME_HOME/bin:$PATH"


3) Config Flume-env.sh

Create a file flume-env.sh in /usr/lib/flume/apache-flume-1.6.0-bin/conf with the following content

export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
export FLUME_CLASS_PATH=/usr/lib/flume/apache-flume-1.6.0-bin/lib/flume-sources-1.0-SNAPSHOT.jar

Note :  choose your JAVA HOME installation folder path according to your environment.


4) Test flume-ng :

cd $FLUME_HOME

flume-ng --help

if you get the flume help command list, then all the variables set correctly.  Good to proceed with agent config.


5) Flume agent config:

Save this file in /usr/lib/flume/apache-flume-1.6.0-bin/conf directory as "example.conf"

# example.conf: A single-node Flume configuration

# Name the components on this agent "Agent1"
agent1.sources = src1
agent1.sinks = sink1
agent1.channels = ch1

# Describe/configure the source
agent1.sources.src1.type = netcat
agent1.sources.src1.bind = localhost
agent1.sources.src1.port = 44444

# Describe the sink
agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory
agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 1000
agent1.channels.ch1.transactionCapacity = 100

# Bind the source and sink to the channel
agent1.sources.src1.channels = ch1
agent1.sinks.sink1.channel = ch1


5) Start the flume agent:

flume-ng agent -n agent1  --conf $FLUME_HOME/conf -f $FLUME_HOME/conf/example.conf -Dflume.root.logger=DEBUG,console

Keep this ssh terminal open.

6) Time to test the the flume flow

Open a new ssh session and connect telnet

root@vm1:~# telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
OK
this is my first msg
OK

Now you should see the message in another terminal.

Congrats you have completed your first Flume implementation.