org.locationtech.geowave.core.ingest.hdfs.mapreduce

Interface IngestFromHdfsPlugin<I,O>

Type Parameters:

I - the type for intermediate data, it must match the type supported by the Avro schema

O - the type that represents each data entry being ingested

All Superinterfaces:

GeoWaveAvroSchemaProvider, IndexProvider

All Known Implementing Classes:

AbstractLocalIngestWithMapper, AbstractSimpleFeatureIngestPlugin, GDELTIngestPlugin, GeoLifeIngestPlugin, GeoWaveAvroIngestPlugin, GpxIngestPlugin, Stanag4676IngestPlugin, TdriveIngestPlugin, TwitterIngestPlugin

public interface IngestFromHdfsPlugin<I,O>
extends IndexProvider, GeoWaveAvroSchemaProvider

This is the main plugin interface for ingesting intermediate data into Geowave that has previously been staged in HDFS. Although both of the available map-reduce ingestion techniques can be implemented (one that simply uses the mapper only, and another that uses the reducer as well), the framework will choose only one based on this plugin's preference, so it is unnecessary to implement both (in this case returning null is expected if its not implemented).

Method Summary

Modifier and Type	Method and Description
IngestWithMapper<I,O>	ingestWithMapper() An implementation of ingestion that can be persisted to a mapper within the map-reduce job configuration to perform an ingest of data into GeoWave from intermediate data
IngestWithReducer<I,?,?,O>	ingestWithReducer() An implementation of ingestion that can be persisted to a mapper and reducer within the map-reduce job configuration to aggregate intermediate data by defined keys within a reducer and perform an ingest of data into GeoWave from the key-value pairs emitted by the mapper.
boolean	isUseReducerPreferred() Returns a flag indicating to the ingestion framework whether it should try to use the ingestWithMapper() implementation or the ingestWithReducer() implementation in the case that both implementations are non-null.

Methods inherited from interface org.locationtech.geowave.core.store.ingest.IndexProvider

getRequiredIndices

Methods inherited from interface org.locationtech.geowave.core.ingest.avro.GeoWaveAvroSchemaProvider

getAvroSchema

Method Detail

isUseReducerPreferred

boolean isUseReducerPreferred()

Returns a flag indicating to the ingestion framework whether it should try to use the ingestWithMapper() implementation or the ingestWithReducer() implementation in the case that both implementations are non-null.

Returns:

If true, the framework will use ingestWithReducer() and only fall back to ingestWithMapper() if necessary, otherwise the behavior will be the reverse

ingestWithMapper

IngestWithMapper<I,O> ingestWithMapper()

An implementation of ingestion that can be persisted to a mapper within the map-reduce job configuration to perform an ingest of data into GeoWave from intermediate data

Returns:

The implementation for ingestion with only a mapper

ingestWithReducer

IngestWithReducer<I,?,?,O> ingestWithReducer()

An implementation of ingestion that can be persisted to a mapper and reducer within the map-reduce job configuration to aggregate intermediate data by defined keys within a reducer and perform an ingest of data into GeoWave from the key-value pairs emitted by the mapper.

Returns:

The implementation for ingestion with a mapper and reducer. It is important to provide the correct concrete implementation of Key and Value classes within the appropriate generics because the framework will use reflection to set the key and value classes for map-reduce.