Create Custom Sink For Flume Sources

Steps To write Azure Sink

1.   Create plugins.d Dir in Flume_HOME(usr/lib/flume)

2     In plugins.d  dir create sink(any name  like azuresink)

3.    In azuresink dir create  three directories

a)lib ----it includes sink class ,one which you have written for custom sink

b)libext ---it includes dependencies of Sink class (all jar needed to run sink class)

c)native  contains dependencies .iso file.

     4. copy the sink jar in /usr/hdp/2.2.6.0-2800/flume/lib    (In HDP 2.2)

          In cloudera there is no need to paste it here.

Bulk Loading Data into HBase Using Mapreduce

If you use BulkLoads with HBase, your workflow is similar to the following:

1. Extract your data from its existing source. For instance, if your data is in a MySQL database, you might run the mysqldump command. The process you use depends on your data. If your data is already in TSV or CSV format, skip this step and use the included ImportTsv utility to process your data into HFiles. See theImportTsv documentation for details.
2. Process your data into HFile format. See http://hbase.apache.org/book/hfile_format.html for details about HFile format. Usually you use a MapReduce job for the conversion, and you often need to write the Mapper yourself because your data is unique. The job must to emit the row key as the Key, and either a KeyValue, a Put, or a Delete as the Value. The Reducer is handled by HBase; configure it using HFileOutputFormat.configureIncrementalLoad() and it does the following:
   • Inspects the table to configure a total order partitioner
   • Uploads the partitions file to the cluster and adds it to the DistributedCache
   • Sets the number of reduce tasks to match the current number of regions
   • Sets the output key/value class to match HFileOutputFormat requirements
   • Sets the Reducer to perform the appropriate sorting (either KeyValueSortReducer or PutSortReducer)
3. One HFile is created per region in the output folder. Input data is almost completely re-written, so you need available disk space at least twice the size of the original data set. For example, for a 100 GB output from mysqldump, you should have at least 200 GB of available disk space in HDFS. You can delete the original input file at the end of the process.
4. Load the files into HBase. Use the LoadIncrementalHFiles command (more commonly known as the completebulkload tool), passing it a URL that locates the files in HDFS. Each file is loaded into the relevant region on the RegionServer for the region. You can limit the number of versions that are loaded by passing the --versions= N option, where N is the maximum number of versions to include, from newest to oldest (largest timestamp to smallest timestamp).
   If a region was split after the files were created, the tool automatically splits the HFile according to the new boundaries. This process is inefficient, so if your table is being written to by other processes, you should load as soon as the transform step is done.

Source :Cloudera

You can get the complete code on my github repository :

https://github.com/RamanandJaiswal/Ramanand-solutions.git