How to build a full flume interceptor by a non java developer

Building a flume interceptor does not look too complex. You can find some examples, howtos or cookbooks around. The issue is that they all explain the specificities of the interceptor, leaving a python/perl guy like me in the dark about maven, classpaths or imports. This posts aims to correct this. This will describe the bare minimum to get it working, but I link to more documentation if you want to go deeper. You can find the code on github as well.

The interceptor itself

You need to create a directory structure, in which to write the following java code. It needs to be under src/main/java/com/example/flume/interceptors/ in this example, for an interceptor named com.example.flume.interceptor.eventTweaker.

package com.example.flume.interceptors;

import java.util.List;
import java.util.Map;

import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.interceptor.Interceptor;

import org.apache.log4j.Logger;

public class eventTweaker implements Interceptor {

  private static final Logger LOG = Logger.getLogger(eventTweaker.class);

  // private means that only Builder can build me.
  private eventTweaker() {}

  public void initialize() {}

  public Event intercept(Event event) {

    Map<String, String> headers = event.getHeaders();

    // example: add / remove headers
    if (headers.containsKey("lice")) {
      headers.put("shampoo", "antilice");

    // example: change body
    String body = new String(event.getBody());
    if (body.contains("injuries")) {
      try {
      } catch ( e) {
        // drop event completely
        return null;

    return event;

  public List<Event> intercept(List<Event> events) {
    for (Event event:events) {
    return events;

  public void close() {}

  public static class Builder implements Interceptor.Builder {

    public Interceptor build() {
      return new eventTweaker();

    public void configure(Context context) {}

Maven configuration

To build the above, you need a pom.xml, to be put at the root of the aforementioned tree. The following is bare minimum which would probably make any java developer cringe, but It Works™.

<project xmlns="" xmlns:xsi=""

 <name>Maven Central</name>
 <name>Cloudera CDH</name>




Once done, you can just type

mvn package

and a jar will be created for you under the target directory.

Tree example

After compilation, here is how my directory structure looks like:

├── pom.xml
├── src
│   └── main
│       └── java
│           └── com
│               └── example
│                   └── flume
│                       └── interceptor
│                           └──
└── target
    ├── classes
    │   └── com
    │       └── example
    │           └── flume
    │               └── interceptors
    │                   ├── eventTweaker$1.class
    │                   ├── eventTweaker$Builder.class
    │                   └── eventTweaker.class
    ├── maven-archiver
    │   └──
    └── eventTweaker-1.0.jar

Jar destination

You need to put the jar (in this case target/eventTweaker-1.0.jar) in the flume classpath for it to be loaded. You can do so in 2 ways:

  1. By editing the FLUME_CLASSPATH variable in the file
  2. By adding it under the $FLUME_HOME/plugins.d directory. In this directory, put your jar at  your_interceptor_name/lib/yourinterceptor.jar and voilà, it will be loaded automagically.

Flume configuration


You can see log statements in the usual flume.log file, probably under /var/log/flume-ng/flume.log


This is the easiest bit. On your source definition, you just have to add an interceptor name and class:


6 thoughts on “How to build a full flume interceptor by a non java developer

  1. In the paragraph “The interceptor itself” you specify a path : “src/main/com/[…]”.
    When you show the structure that you have, there is a “java” folder in addition.
    Without this “java” folder, the maven command work (“success”) but there is noghing in the jar file produced.
    Without this mistake, your tuto is really helpful, clear and fast to do, thank you for it !

  2. Pingback: error compiling java -use source 5

  3. Great!, I have a Java Client that send bytes[] to flume and my interceptor should transform to json, but this not happen, I was trying to see flume log but all its ok, there is an option to monitoring in realtime my object and interceptor? Thanks ans great Post

  4. hello boss,
    My flume interceptor event is not taking whole json takes only half the string and thorws error while json parsing.
    PFB the issue.Kindly help.
    {“parent”:””,”caused_by”:””,”watch_list”:””,”upon_reject”:”cancel”,”sys_updated_on”:”2017-01-11 01:00:07″,”u_escalate”:”false”,”u_escalation_reason”:””,”approval_history”:””,”skills”:””,”number”:”INC0276556″,”u_geo”:””,”state”:”7″,”u_oasis_reservation”:””,”sys_created_by”:”bijayak”,”u_type_of_impact”:””,”knowledge”:”false”,”order”:””,”u_country”:””,”cmdb_ci”:{“link”:”″,”value”:”383ed8c86f3c2200f28cfee09d3ee4e6″},”contract”:””,”impact”:”3″,”active”:”false”,”work_notes_list”:””,”priority”:”5″,”sys_domain_path”:”/”,”business_duration”:”1970-01-01 00:00:00″,”group_list”:””,”approval_set”:””,”short_description”:”Outlook not responding on BLR-HRZ view”,”correlation_display”:””,”work_start”:””,”additional_assignee_list”:””,”notify”:”1″,”sys_class_name”:”incident”,”closed_by”:{“link”:”″,”value”:”b0b76aaf4f456a04ec8da3928110c766″},”follow_up”:””,”parent_incident”:””,”reassignment_count”:”0″,”assigned_to”:{“link”:”″,”value”:”b0b76aaf4f456a04ec8da3928110c766″},”u_region”:””,”u_users_impacted”:”me”,”sla_due”:””,”comments_and_work_notes”:””,”u_category”:{“link”:””,”value”:”417c30486f7c2200f28cfee09d3ee4af”},”escalation”:”0″,”u_related_request”:””,”upon_approval”:”proceed”,”correlation_id”:””,”u_region_group”:””,”made_sla”:”true”,”u_escalate_updated”:””,”child_incidents”:”0″,”u_outage”:”false”,”resolved_by”:{“link”:”″,”value”:”b0b76aaf4f456a04ec8da3928110c766″},”sys_updated_by”:”system”,”opened_by”:{“link”:”″,”value”:”b0b76aaf4f456a04ec8da3928110c766″},”user_input”:””,”sys_creat
    2017-04-28 07:35:35,912 (pool-3-thread-1) [ERROR – com.example.flume.interceptors.eventTweaker.intercept(] exception Unterminated string at 2049 [character 2050 line 1]
    2017-04-28 07:35:35,912 (pool-3-thread-1) [INFO – com.example.flume.interceptors.eventTweaker.intercept(] Received this log message that is not formatted in json:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s