org.locationtech.geowave.core.store.api

Interface DataStore

All Known Subinterfaces:

MapReduceDataStore

All Known Implementing Classes:

AccumuloDataStore, BaseDataStore, BaseMapReduceDataStore, CassandraDataStore, DynamoDBDataStore, FileSystemDataStore, HBaseDataStore, KuduDataStore, RedisDataStore, RocksDBDataStore

public interface DataStore

A DataStore can both ingest and query data based on persisted indices and data type adapters. When the data is ingested it is explicitly given an index and a data type adapter which is then persisted to be used in subsequent queries. Also, implicitly statistics are maintained associated with all data ingested. These statistics can be queried. Furthermore, aggregations can be applied directly to the data which are similar to statistics, but are more dynamic in that any query criteria can be applied as the input of the aggregation. Data stores that support server-side processing will run the aggregation within the scope of iterating through the results for additional efficiency.

Here is a simple snippet of pseudocode showing how a data store can be used to store and retrieve your data.

 
  DataStore store = DataStoreFactory.createDataStore(<data store options>);
 	store.addType(<my data type>, <my index>);
  try(Writer writer = store.createWriter()){
    //write data
    writer.writer(<data);
  }

  //this just queries everything
  try(CloseableIterator it = store.query(QueryBuilder.newBuilder().build())){
    while(it.hasNext()){
      //retrieve results matching query criteria and do something
    	it.next();
    }
  }
 
 

Method Summary

Modifier and Type	Method and Description
void	addEmptyStatistic(Statistic<?>... statistic) Add a statistic to the data store.
void	addIndex(Index index) Add an index to the data store.
void	addIndex(String typeName, Index... indices) Add new indices for the given type.
void	addStatistic(Statistic<?>... statistic) Add a statistic to the data store.
<T> void	addType(DataTypeAdapter<T> dataTypeAdapter, Index... initialIndices) Add this type to the data store.
<T> void	addType(DataTypeAdapter<T> dataTypeAdapter, List<Statistic<?>> statistics, Index... initialIndices) Add this type to the data store with the given statistics.
<T> void	addType(DataTypeAdapter<T> dataTypeAdapter, VisibilityHandler visibilityHandler, List<Statistic<?>> statistics, Index... initialIndices) Add this type to the data store with the given statistics and visibility handler.
<P extends Persistable,R,T> R	aggregate(AggregationQuery<P,R,T> query) Perform an aggregation on the data and just return the aggregated result.
<V extends StatisticValue<R>,R> V	aggregateStatistics(StatisticQuery<V,R> query) Get a single statistical result that matches the given query criteria
void	copyTo(DataStore other) copy all data from this store into a specified other store
void	copyTo(DataStore other, Query<?> query) copy the subset of data matching this query from this store into a specified other store
<T> Writer<T>	createWriter(String typeName) Returns an index writer to perform batched write operations for the given data type name.
<T> Writer<T>	createWriter(String typeName, VisibilityHandler visibilityHandler) Returns an index writer to perform batched write operations for the given data type name.
<T> boolean	delete(Query<T> query) Delete all data in this data store that matches the query parameter.
void	deleteAll() Delete ALL data and ALL metadata for this datastore.
default <V extends StatisticValue<R>,R> CloseableIterator<org.apache.commons.lang3.tuple.Pair<ByteArray,R>>	getBinnedStatisticValues(Statistic<V> stat) Returns all of the statistic values of this stat as well as the associated bin.
<V extends StatisticValue<R>,R> CloseableIterator<org.apache.commons.lang3.tuple.Pair<ByteArray,R>>	getBinnedStatisticValues(Statistic<V> stat, BinConstraints binConstraints) The statistic values of this stat as well as the associated bin.
<V extends StatisticValue<R>,R> DataTypeStatistic<V>	getDataTypeStatistic(StatisticType<V> statisticType, String typeName, String tag) Gets the statistic that is being tracked for the data type, statistic type, and tag specified.
DataTypeStatistic<?>[]	getDataTypeStatistics(String typeName) Gets all of the statistics that are being tracked on the provided data type adapter.
<V extends StatisticValue<R>,R> FieldStatistic<V>	getFieldStatistic(StatisticType<V> statisticType, String typeName, String fieldName, String tag) Gets the statistic that is being tracked for the data type, field, statistic type, and tag specified.
FieldStatistic<?>[]	getFieldStatistics(String typeName, String fieldName) Gets all of the statistics that are being tracked on the provided type/field pair.
Index	getIndex(String indexName) Get a particular index by its index name.
<V extends StatisticValue<R>,R> IndexStatistic<V>	getIndexStatistic(StatisticType<V> statisticType, String indexName, String tag) Gets the statistic that is being tracked for the index, statistic type, and tag specified.
IndexStatistic<?>[]	getIndexStatistics(String indexName) Gets all of the statistics that are being tracked on the provided index.
Index[]	getIndices() Get the indices that have been used within this data store.
Index[]	getIndices(String typeName) Get the indices that have been used within this data store for a particular type.
default <V extends StatisticValue<R>,R> R	getStatisticValue(Statistic<V> stat) The statistic value of this stat (if multiple bins match, it will automatically aggregate the resulting values together).
<V extends StatisticValue<R>,R> R	getStatisticValue(Statistic<V> stat, BinConstraints binConstraints) The statistic value of this stat (if multiple bins match, it will automatically aggregate the resulting values together).
DataTypeAdapter<?>	getType(String typeName) Get the data type adapter with the given type name from the data store.
DataTypeAdapter<?>[]	getTypes() Get all the data type adapters that have been used within this data store
<T> void	ingest(String inputPath, Index... index) Ingest from path.
<T> void	ingest(String inputPath, IngestOptions<T> options, Index... index) Ingest from path with options.
<T> CloseableIterator<T>	query(Query<T> query) Returns all data in this data store that matches the query parameter.
ResultSet	query(String queryStr, String... authorizations) Perform a query using the GeoWave Query Language (GWQL).
<V extends StatisticValue<R>,R> CloseableIterator<V>	queryStatistics(StatisticQuery<V,R> query) Get data statistics that match the given query criteria
void	recalcStatistic(Statistic<?>... statistic) Force a recomputation of the stats
void	removeIndex(String indexName) remove an index completely for all types.
void	removeIndex(String typeName, String indexName) remove an index for the given type.
void	removeStatistic(Statistic<?>... statistic) Remove statistics from the data store.
void	removeType(String typeName) Remove all data and statistics associated with the given type.

Method Detail

ingest

<T> void ingest(String inputPath,
                Index... index)

Ingest from path. If this is a directory, this method will recursively search for valid files to ingest in the directory. This will iterate through registered IngestFormatPlugins to find one that works for a given file. The applicable ingest format plugin will choose the DataTypeAdapter and may even use additional indices than the one provided.

Parameters:

inputPath - The path for data to read and ingest into this data store

index - The indexing approach to use.

ingest

<T> void ingest(String inputPath,
                IngestOptions<T> options,
                Index... index)

Ingest from path with options. If this is a directory, this method will recursively search for valid files to ingest in the directory. The applicable ingest format plugin will choose the DataTypeAdapter and may even use additional indices than the one provided.

Parameters:

inputPath - The path for data to read and ingest into this data store

options - a set of available options for ingesting from a URL

index - The configuration information for the primary index to use.

query

<T> CloseableIterator<T> query(Query<T> query)

Returns all data in this data store that matches the query parameter. All data that matches the query will be returned as an instance of the native data type. The Iterator must be closed when it is no longer needed - this wraps the underlying scanner implementation and closes underlying resources.

Parameters:

query - data constraints for the query and additional options for processing the query

Returns:

An iterator on all results that match the query. The iterator implements Closeable and it is best practice to close the iterator after it is no longer needed.

query

ResultSet query(String queryStr,
                String... authorizations)

Perform a query using the GeoWave Query Language (GWQL).

Parameters:

queryStr - the GWQL query to perform

authorizations - the authorizations to use for the query

Returns:

the set of results that match the given query string

aggregate

<P extends Persistable,R,T> R aggregate(AggregationQuery<P,R,T> query)

Perform an aggregation on the data and just return the aggregated result. The query criteria is very similar to querying the individual entries except in this case it defines the input to the aggregation function, and the aggregation function produces a single result. Examples of this might be simply counting matched entries, producing a bounding box or other range/extent for matched entries, or producing a histogram.

Parameters:

query - the Aggregation Query, use AggregationQueryBuilder or its extensions to create

Returns:

the single result of the aggregation

getType

DataTypeAdapter<?> getType(String typeName)

Get the data type adapter with the given type name from the data store.

Parameters:

typeName - the name of the type to get

Returns:

The data type adapter with the given name, or null if it couldn't be found

getTypes

DataTypeAdapter<?>[] getTypes()

Get all the data type adapters that have been used within this data store

Returns:

An array of the types used within this datastore.

addEmptyStatistic

void addEmptyStatistic(Statistic<?>... statistic)

Add a statistic to the data store. The initial value of the statistic will not be calculated and if there is existing relevant data, this statistic will not be accurate without forcing a calculation. If instead it is not desire-able to calculate on add use addStatistic instead.

Parameters:

statistics - the statistics to add

addStatistic

void addStatistic(Statistic<?>... statistic)

Add a statistic to the data store. The initial value of the statistic will be calculated after being added. If this calculation is not desired use addEmptyStatistic instead.

Parameters:

statistics - the statistics to add

removeStatistic

void removeStatistic(Statistic<?>... statistic)

Remove statistics from the data store.

Parameters:

statistic - the statistics to remove

recalcStatistic

void recalcStatistic(Statistic<?>... statistic)

Force a recomputation of the stats

Parameters:

statistic - the statistics to recompute

getDataTypeStatistics

DataTypeStatistic<?>[] getDataTypeStatistics(String typeName)

Gets all of the statistics that are being tracked on the provided data type adapter.

Parameters:

typeName - the data type adapter to get the statistics for

Returns:

An array of all the statistics that are being tracked on the provided data type adapter. Note this is the descriptors of the statistics, not the values.

getDataTypeStatistic

<V extends StatisticValue<R>,R> DataTypeStatistic<V> getDataTypeStatistic(StatisticType<V> statisticType,
                                                                          String typeName,
                                                                          @Nullable
                                                                          String tag)

Gets the statistic that is being tracked for the data type, statistic type, and tag specified.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

statisticType - the statistic type for the statistic to get

typeName - the data type name to get the statistic for

tag - the tag of the statistic, if not specified, a tag will be inferred

Returns:

the statistic, or null if no statistic matches the criteria

getIndexStatistics

IndexStatistic<?>[] getIndexStatistics(String indexName)

Gets all of the statistics that are being tracked on the provided index.

Parameters:

indexName - the index name to retrieve statistics for

Returns:

An array of all the statistics that are being tracked on the provided index. Note this is the descriptors of the statistics, not the values.

getIndexStatistic

<V extends StatisticValue<R>,R> IndexStatistic<V> getIndexStatistic(StatisticType<V> statisticType,
                                                                    String indexName,
                                                                    @Nullable
                                                                    String tag)

Gets the statistic that is being tracked for the index, statistic type, and tag specified.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

statisticType - the statistic type for the statistic to get

indexName -

tag - the tag of the statistic, if not specified, a tag will be inferred

Returns:

the statistic, or null if no statistic matches the criteria

getFieldStatistics

FieldStatistic<?>[] getFieldStatistics(String typeName,
                                       String fieldName)

Gets all of the statistics that are being tracked on the provided type/field pair.

Parameters:

typeName - the data type name to get the statistics for

fieldName - the field name to get the statistics for

Returns:

An array of all the statistics that are being tracked on the provided field. Note this is the descriptors of the statistics, not the values.

getFieldStatistic

<V extends StatisticValue<R>,R> FieldStatistic<V> getFieldStatistic(StatisticType<V> statisticType,
                                                                    String typeName,
                                                                    String fieldName,
                                                                    @Nullable
                                                                    String tag)

Gets the statistic that is being tracked for the data type, field, statistic type, and tag specified.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

statisticType - the statistic type for the statistic to get

typeName - the data type name to get the statistic for

fieldName -

tag - the tag of the statistic, if not specified, a tag will be inferred

Returns:

the statistic, or null if no statistic matches the criteria

getStatisticValue

default <V extends StatisticValue<R>,R> R getStatisticValue(Statistic<V> stat)

The statistic value of this stat (if multiple bins match, it will automatically aggregate the resulting values together). For statistics with bins, it will always aggregate all bins.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

stat - the statistic to get the value for

Returns:

the statistic's value, aggregated together if there are multiple matching values.

getStatisticValue

<V extends StatisticValue<R>,R> R getStatisticValue(Statistic<V> stat,
                                                    BinConstraints binConstraints)

The statistic value of this stat (if multiple bins match, it will automatically aggregate the resulting values together).

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

stat - the statistic to get the value for

binConstraints - the bin(s) to get the value for based on the constraints

Returns:

the statistic's value, aggregated together if there are multiple matching values.

getBinnedStatisticValues

default <V extends StatisticValue<R>,R> CloseableIterator<org.apache.commons.lang3.tuple.Pair<ByteArray,R>> getBinnedStatisticValues(Statistic<V> stat)

Returns all of the statistic values of this stat as well as the associated bin. It will return each individual match as a bin-value pair.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

stat - the statistic to get the value for

Returns:

the statistic bin-value pairs, if there are multiple matching values which should only be the case for different bins it will return each individual value. It will return an empty iterator if there are no matching values.

getBinnedStatisticValues

<V extends StatisticValue<R>,R> CloseableIterator<org.apache.commons.lang3.tuple.Pair<ByteArray,R>> getBinnedStatisticValues(Statistic<V> stat,
                                                                                                                             BinConstraints binConstraints)

The statistic values of this stat as well as the associated bin. If multiple bins match, it will return each individual match as a bin-value pair.

Type Parameters:

V - the StatisticValue implementation of the statistic

R - the raw value type of the statistic

Parameters:

stat - the statistic to get the value for

binConstraints - the bin(s) to get the value for based on the constraints

Returns:

the statistic bin-value pairs, if there are multiple matching values which should only be the case for different bins it will return each individual value. It will return an empty iterator if there are no matching values.

queryStatistics

<V extends StatisticValue<R>,R> CloseableIterator<V> queryStatistics(StatisticQuery<V,R> query)

Get data statistics that match the given query criteria

Parameters:

query - the query criteria, use StatisticQueryBuilder or its extensions and if you're interested in a particular common statistics type use StatisticsQueryBuilder.factory()

Returns:

An array of statistics that result from the query

aggregateStatistics

<V extends StatisticValue<R>,R> V aggregateStatistics(StatisticQuery<V,R> query)

Get a single statistical result that matches the given query criteria

Parameters:

query - the query criteria, use StatisticQueryBuilder or its extensions and if you're interested in a particular common statistics type use StatisticsQueryBuilder.factory()

Returns:

If the query does not define that statistics type it will return null as aggregation only makes sense within a single type, otherwise aggregates the results of the query into a single result that is returned

addIndex

void addIndex(Index index)

Add an index to the data store.

Parameters:

index - the index to add

getIndices

Index[] getIndices()

Get the indices that have been used within this data store.

Returns:

all indices used within this datastore

getIndices

Index[] getIndices(String typeName)

Get the indices that have been used within this data store for a particular type. If data type name is null it will return all indices.

Parameters:

the - data type name

Returns:

An array of the indices for a given data type.

getIndex

Index getIndex(String indexName)

Get a particular index by its index name. If one doesn't exist it will return null.

Parameters:

indexName - the index name for which to retrieve an index

Returns:

The index matching the specified index name or null if it doesn't exist

copyTo

void copyTo(DataStore other)

copy all data from this store into a specified other store

Parameters:

other - the other store to copy data into

copyTo

void copyTo(DataStore other,
            Query<?> query)

copy the subset of data matching this query from this store into a specified other store

Parameters:

other - the other store to copy data into

query - a query to select which data to copy - use QueryBuilder or its extension to create

addIndex

void addIndex(String typeName,
              Index... indices)

Add new indices for the given type. If there is data in other indices for this type, for consistency it will need to copy all of the data into the new indices, which could be a long process for lots of data.

Parameters:

typeName - the type

indices - the new indices to add

removeIndex

void removeIndex(String indexName)
          throws IllegalStateException

remove an index completely for all types. If this is the last index for any type it throws an illegal state exception, expecting the user to remove the type before removing the index to protect a user from losing any reference to their data unknowingly for a type.

Parameters:

indexName - the index

Throws:

IllegalStateException - if this is the last index for a type, remove the type first

removeIndex

void removeIndex(String typeName,
                 String indexName)
          throws IllegalStateException

remove an index for the given type. If this is the last index for that type it throws an illegal state exception, expecting the user to remove the type before removing the index to protect a user from losing any reference to their data unknowingly for a type.

Parameters:

typeName - the type

indexName - the index

Throws:

IllegalStateException - if this is the last index for a type, remove the type first

removeType

void removeType(String typeName)

Remove all data and statistics associated with the given type.

Parameters:

typeName - the type

delete

<T> boolean delete(Query<T> query)

Delete all data in this data store that matches the query parameter.

Statistics are updated as required.

Parameters:

query - the query criteria to use for deletion

Returns:

true on success

deleteAll

void deleteAll()

Delete ALL data and ALL metadata for this datastore. This is provided for convenience as a simple way to wipe a datastore cleanly, but don't be surprised if everything is gone.

addType

<T> void addType(DataTypeAdapter<T> dataTypeAdapter,
                 Index... initialIndices)

Add this type to the data store. This only needs to be called one time ever per type.

Parameters:

dataTypeAdapter - the data type adapter for this type that is used to read and write GeoWave entries

initialIndices - the initial indexing for this type, in the future additional indices can be added

addType

<T> void addType(DataTypeAdapter<T> dataTypeAdapter,
                 List<Statistic<?>> statistics,
                 Index... initialIndices)

Add this type to the data store with the given statistics. This only needs to be called one time ever per type.

Parameters:

dataTypeAdapter - the data type adapter for this type that is used to read and write GeoWave entries

statistics - the initial set of statistics that will be used with this adapter

initialIndices - the initial indexing for this type, in the future additional indices can be added

addType

<T> void addType(DataTypeAdapter<T> dataTypeAdapter,
                 VisibilityHandler visibilityHandler,
                 List<Statistic<?>> statistics,
                 Index... initialIndices)

Add this type to the data store with the given statistics and visibility handler. This only needs to be called one time ever per type.

Parameters:

dataTypeAdapter - the data type adapter for this type that is used to read and write GeoWave entries

visibilityHandler - the visibility handler for the adapter entries

statistics - the initial set of statistics that will be used with this adapter

initialIndices - the initial indexing for this type, in the future additional indices can be added

createWriter

<T> Writer<T> createWriter(String typeName)

Returns an index writer to perform batched write operations for the given data type name. It assumes the type has already been used previously or added using addType and assumes one or more indices have been provided for this type.

Parameters:

typeName - the type

Returns:

a writer which can be used to write entries into this datastore of the given type

createWriter

<T> Writer<T> createWriter(String typeName,
                           VisibilityHandler visibilityHandler)

Returns an index writer to perform batched write operations for the given data type name. It assumes the type has already been used previously or added using addType and assumes one or more indices have been provided for this type.

Parameters:

typeName - the type

visibilityHandler - the visibility handler for newly written entries

Returns:

a writer which can be used to write entries into this datastore of the given type