GeoWave Developer Guide

This guide focuses on the development side of GeoWave. It also serves as a deep dive into some of the inner workings of GeoWave. The target audience for this guide are GeoWave developers and developers who wish to use GeoWave as part of another software package.

This guide assumes the following:

GeoWave development requires the following components:

Users have two options for retrieving the GeoWave source code: either by cloning the repository or by directly downloading the code repository as a ZIP archive.

The recommended Integrated Development Environment (IDE) for GeoWave is Eclipse. This section will walk you through importing the GeoWave Maven projects into the Eclipse IDE.

Using the Eclipse Maven M2Eclipse plugin, we can import Maven projects into Eclipse. When importing Maven projects, Eclipse will automatically resolve and download dependencies listed in the pom.xml file for each project.

To build the project source, navigate to the root directory of the GeoWave project using a command-line tool and execute the following maven command:

After executing the command, Maven will search for all of the projects that need to be built and begin the build process. The initial output of the command should look something like the following:

The build process can take several minutes, but once this is completed, the compiled artifacts for each project will be installed to your local Maven repository. They will also be available in each project’s target directory.

By default, integration tests are not run as part of the normal build process. This is because integration tests can be run for a multitude of key/value store backends, and tests for a single backend can take a significant amount of time to complete. Usually these integration tests are run through GitHub Actions, but it can be useful to run them locally when working on code that could potentially impact one of the tests.

Integration tests are all written in the geowave-tests project and are run using a series of Maven profiles. The following table shows the various profiles available along with a description of the tests that are run:

In order to use one of these profiles to run integration tests, use the same command that was used to build the source, but add the appropriate profile to the end. For example, if you wanted to run integration tests for RocksDB, the command would look like the following:

If you have already built GeoWave, you can skip straight to the integration tests:

The Python bindings for GeoWave (pygw) use a different build process than the Java component. The Python source code for pygw can be found in the python/src/main/python directory. In order to install pygw from source, you will need Python 3 (up to Python 3.7) and Virtualenv.

GeoWave documentation consists of several different parts, the main documentation, which includes this guide, the Javadocs, and the Python bindings documentation.

We have support for building both the GeoWave JAR artifacts and RPMs from Docker containers. This capability is useful for a number of different situations:

If building artifacts using Docker containers interests you, check out the README in deploy/packaging/docker.

GeoWave artifacts can be packaged into a single JAR that can be used to execute CLI commands by running the following Maven command from the GeoWave root directory:

After the packaging process is complete, the resulting JAR will be available in the deploy/target directory with a name like geowave-deploy-<version>-geoserver.jar. To use this jar for CLI commands you can execute it using the following java command:

As you can see, using GeoWave in this way can be fairly cumbersome. To make things easier, this command can be wrapped up in an alias.

After the alias has been created, you will be able to use the GeoWave CLI with the geowave command. For a full list of these commands, please see the GeoWave CLI Appendix.

GeoWave artifacts can be packaged into a single JAR to be used in a GeoServer installation by running the following Maven command from the GeoWave root directory:

After the packaging process is complete, the resulting JAR will be available in the deploy/target directory with a name like geowave-deploy-<version>-geoserver.jar. To use this jar with a GeoServer installation, simply copy it to the WEB-INF/lib directory of GeoServer’s installation and restart the web service that GeoServer is running on.

GeoWave artifacts can be packaged into a JAR to be used by Accumulo for server-side operations by running the following Maven command from the GeoWave root directory:

After the packaging process is complete, the resulting JAR will be available in the deploy/target directory with a name like geowave-deploy-<version>-accumulo.jar. See Accumulo Configuration in the User Guide for more information about using this jar.

GeoWave artifacts can be packaged into a coprocessor JAR for HBase server-side operations by running the following Maven command from the GeoWave root directory:

After the packaging process is complete, the resulting JAR will be available in the deploy/target directory with a name like geowave-deploy-<version>-hbase.jar. In order to use this jar, copy it to an HDFS location that is accessible to HBase. When configuring the GeoWave HBase data store (either through the GeoServer plugin or the CLI), set the coprocessorJar option to the HDFS location of the jar.

Standalone installers for Linux, Mac, and Windows can be built using Install4J.

Several things need to be done in order to successfully build the standalone installers, each of which are outlined below.

Pull requests must be done though a forked repository.

To create a new fork, just click the the "Fork" button at the top of the GeoWave GitHub page. You can now submit pull requests from your working branch on your fork directly to the master branch on the locationtech repository.

GeoWave uses a Maven plugin formatter to keep all of our code standardized. You should run a Maven install immediately prior to committing and pushing changes.

Prior to submitting a pull request, please squash down your commits into one. This will help keep our commit history clean and will cut down on "in progress commits" that don’t relay any helpful information in the future.

All contributors must sign the Eclipse Contributor Agreement and sign off all commits by using the --signoff option of the commit command.

All pull request contributions to this project will be released under the Apache 2.0 license.

Software source code previously released under an open source license and then modified by NGA staff is considered a "joint work" (see 17 USC 101); it is partially copyrighted, partially public domain, and as a whole is protected by the copyrights of the non-government authors and must be released according to the terms of the original open source license.

The core of the GeoWave architecture concept is getting data in (Ingest), and pulling data out (Query). This is accomplished by using data adapters and indices. As discussed in the GeoWave Overview, data adapters describe the available fields in a data type and are used to transform data from the base type into a format that is optimized for GeoWave. An index is used to determine the organization and storage of the converted data so that it can be efficiently queried. There are two types of data persisted in the system: indexed data and metadata. Indexed data is the data (such as vector attributes and geometries) that has been converted to the GeoWave format by an adapter and stored using the index. Metadata contains all of the information about the state of the data store, such as the adapters, indices, and any statistics that have been created for a type. The intent is to store all of the information needed for data discovery and retrieval in the database. This means that an existing data store isn’t tied to a bit of configuration on a particular external server or client but instead is “self-describing.”

The following diagram describes the default structure of indexed data in a GeoWave data store.

These structures are described by two interfaces: GeoWaveKey and GeoWaveValue. It is up to the data store implementation to determine how to use these structures to ultimately store GeoWave data, so the final structure may vary between implementations.

GeoWave data stores are made up of several different components that each manage different aspects of the system, such as an adapter store, index store, statistics store, etc. Most of the time, directly using these components should not be necessary as most GeoWave tasks can be accomplished through the use of the DataStore interface.

Programmatically, data stores are accessed by using a StoreFactoryOptions implementation for the appropriate key/value store to configure a connection to that store. Once configured with all of the necessary options, the DataStoreFactory can be used to directly create a DataStore instance.

An instance of DataStorePluginOptions can be also be created from the StoreFactoryOptions if direct access to other parts of the data store is needed.

For an example of accessing a data store through the programmatic API, see the Creating Data Stores example.

The way that GeoWave stores data in a way that makes it efficient to query is through the use of Indices. As mentioned in the overview, indices use a given set of dimensions to determine the order in which the data is stored. Indices are composed of two components: a common index model, and an index strategy.

In order to store geometry, attributes, and other information, input data must be converted to a format that is optimized for data discovery. GeoWave provides a DataTypeAdapter interface that handles this conversion process. Implementations that support GeoTools simple feature types as well as raster data are included. When a data adapter is used to ingest data, the adapter and its parameters are persisted as metadata in the GeoWave data store. When the type is queried, the adapter is loaded dynamically in order to translate the GeoWave data back into its native form.

GeoWave statistics are stored as metadata and can be queried for aggregated information about a particular data type, field, or index. Statistics retain the same visibility constraints as the data they are associated with. For example, let’s say there is an data type that has several rows with a visibility expression of A&B, and several more rows with a visibility expression of A&C. If there was count statistic on this data, then there would be two rows in the statistics table, one for the number of rows with the A&B visibility, and another for the number of rows with the A&C visibility.

There are three different types of statistics, each of which extend from a different base statistic class.

Sometimes it is desirable to split up a statistic into several bins using some arbitrary method. Each bin is identified by a unique byte array and contains its own statistic value for all rows that fall into it. GeoWave includes a few binning strategies that cover a majority of simple use cases.

In order to provide as much flexibility as possible to developers, the StatisticBinningStrategy interface has been made available so that new binning strategies can be added as the need arises. This is described in more detail below.

Statistics are composed of two different types of objects that are stored within the GeoWave metadata tables: statistics and the statistic values.

The statistic table contains all of the tracked statistics for the data store. Each row of this table describes one statistic and contains all of the information needed to properly calculate the statistic as data is ingested and deleted.

The following diagram describes the default structure of a statistic in a GeoWave data store.

The values of these statistics are stored separately as GeoWave metadata with a similar structure.

There are two primary ways to get the value of a given statistic. The first and easiest way is to use the Statistic object itself as a parameter to getStatisticValue or getBinnedStatisticValues on the DataStore interface. If the statistic uses a binning strategy, a set of bin constraints can also be supplied to filter down the bins that are returned by the query. Each binning strategy supports different types of constraints, which can be discovered through the supportedConstraintClasses method. These constraint classes can be converted into bin constraints by passing them to the constraints method on the binning strategy. For example, TimeRangeFieldValueBinningStrategy supports Interval as a constraint class. All bins within a given time interval could be queried by passing the result of constraints(interval) to the getStatisticValue method on the DataStore. These methods do not take visibility of rows into account and will get the values for all visibilities by default.

The second way statistic values can be retrieved is to query the statistic by using a StatisticQueryBuilder. A query builder of the appropriate type can be obtained by calling one of the newBuilder static methods on StatisticQueryBuilder with the StatisticType to query. Once all of the query parameters and optional constraints have been set and the query is built, the resulting StatisticQuery object can then be passed to the queryStatistics or aggregateStatistics functions of the DataStore. Each of these functions performs the same query, but outputs different values. The queryStatistics function will return one StatisticValue instance for every bin for each statistic that matched the query parameters, while the aggregateStatistics function will merge all of those values down to a single StatisticValue instance. A statistic query allows you to provide authorizations if the result should be filtered by visibility.

New statistics can be implemented by extending the appropriate statistic type (IndexStatistic, DataTypeStatistic, or FieldStatistic) and implementing a corresponding StatisticValue. It is recommended that a public static STATS_TYPE variable be made available to make the StatisticType of the statistic readily available to users.

New binning strategies can also be added by implementing the StatisticBinningStrategy interface. The binning strategy can use information from the DataTypeAdapter, the raw entry, and the GeoWaveRow(s) that the entry was serialized to in order to determine the bin that should be used. It is also recommended to provide some level of support for constraints that would be relevant to the binning strategy to make it easier for end users to constrain statistics queries.

All statistics and binning strategies are discovered by GeoWave using Service Provider Interfaces (SPI). In order to add new statistics and binning strategies, extend the StatisticsRegistrySPI and make sure the JAR containing both the registry and the statistic/statistic value classes are on the classpath when running GeoWave. For more information on using SPI, see the Oracle documentation.

An example that shows how to add a word count statistic is available in the GeoWave examples project.

In addition to the raw data, the ingest process requires an adapter to translate the native data into a format that can be persisted into the data store, and an index to dictate how the ingested data should be organized. The following diagram shows an overview of the ingest process:

The logic within the ingest process immediately ensures that the index and data adapter are persisted as metadata within the index and adapter stores to support self-described data discovery. In-memory implementations of both of these stores are provided for cases when connections to third-party data stores (e.g., Accumulo, HBase) are undesirable in the ingest process, such as ingesting bulk data in a MapReduce job. Once the adapter and index have been persisted, each data entry gets processed by the adapter to encode the data to a format that’s optimized for GeoWave, and by the index to determine where the data should be stored.

Once the GeoWaveKey and GeoWaveValue are created for an entry, they are combined into a GeoWaveRow and sent to the RowWriter implementation to be written to the data store.

The full list of GeoWave ingest commands can be found in the GeoWave CLI Documentation.

Ingest plugins contain everything that is needed to convert raw data files into a format that is understood by a GeoWave data adapter. Ingest plugins hook into the ingest framework and are decoupled from the data store and index implementations.

When a query is performed, GeoWave extracts index constraints from the provided query filter. These index constraints are then decomposed into a set of range queries according to the index strategy that is used by the index. See the Theory section for information about how ranges are decomposed for multi-dimensional data. These range queries represent the coarse grain filtering of the query.

The query filter is broken down into two types of filters: distributable and client. Distributable filters are filters that operate on the common index data while client filters are filters that operate on the extended data of the feature. Distributable filters are serialized and sent to the data store in order to filter the results of the range queries server-side. An example of a distributable filter is a geometry filter. The index constraints extracted from the geometry filter are generally loser than the actual geometry to simplify the number of range queries that need to be performed. Because of this, results from the range queries must pass through the actual geometry filter to remove any entries that do not match exactly.

All results that pass the distributable filters are then returned to the client which decodes each entry using the data adapter and discards any entries that do not pass the remaining client filters.

Queries are created in GeoWave through the use of query builders. These builders are used to set all the things needed to create a query, such as the type names, indices, authorizations, and query constraints. While the base QueryBuilder can be used as a general way to query data, GeoWave also provides an implementation of the query builder that is specific to vector queries with the VectorQueryBuilder. It also provides a query builder for vector aggregation queries with the VectorAggregationQueryBuilder. These vector query builders provide a constraints factory that has additional constraints that are specific to vector queries, such as CQL filters. See the programmatic API examples for examples of these query builders in action.

Queries can also be filtered and constrained using a GeoWave filter expression. The easiest way to do this is to create an appropriate FieldValue expression based on the field you wish to constrain. GeoWave provides commonly used expressions and predicates for spatial, temporal, numeric, and text field values. Expressions can also be combined to create more complex query filters. Additionally, if no index name is provided to the query builder when using a filter expression, GeoWave will infer the best index based on the fields that are constrained by the filter. The following is an example of a query that uses a GeoWave filter expression:

Another common way of filtering vector data in a query is by using CQL, also known as Common Query Language. CQL makes query filters more human readable and understandable while still maintaining the complexity that is often necessary. The constraints factory that is provided by the VectorQueryBuilder contains a helper function for creating query constraints using a CQL expression. CQL query constraints are used through the programmatic API, the GeoServer plugin, and through the GeoWave Query Lanaguage. CQL query filters are less efficient that GeoWave filter expressions, but can be useful if one of the needed capabilities are not yet implemented by the GeoWave filter expressions. For an overview on using CQL, please refer to the GeoServer tutorials.

In order to simplify queries, GeoWave provides a simple query language that is roughly based on SQL. This is discussed in the User Guide. While the user guide discusses the language from the context of the CLI, it is also possible to execute these queries programmatically through the DataStore interface. For example, the following statement would execute an everything query on the countries type in the example data store:

Querying GeoWave using the GeoWave Query Language will return results in the form of a ResultSet, which is less like the results that would be obtained from a standard GeoWave query (e.g. SimpleFeatures) and more like the results that you would expect from querying a relational database (Rows) in that only the fields and aggregations included in the SELECT statement will be returned.

GeoWave’s gRPC service provides a way for remote gRPC client applications to interact with GeoWave.

Data stores are created by instantiating a StoreFactoryOptions implementation for the data store type you want to create. The following table lists the various options classes for each supported key/value store:

Once the options class has been initialized with all of the desired options, a DataStore can be created using the DataStoreFactory. The following example shows how to create a RocksDB data store:

Spatial and spatial-temporal indices are created by using the appropriate index builder. For spatial indices, the SpatialIndexBuilder can be used, and for spatial-temporal indices, the SpatialTemporalIndexBuilder can be used. Each builder has options that are specific to the type of index being created. Once all options have been set, the index can be created with the createIndex function.

This index can then be added to the data store with the addIndex function. The following example shows how to add these indices to the RocksDB data store created in the previous section:

Data can be ingested into GeoWave by adding a type to a set of indices and then using the Writer interface to write data. The following example creates a FeatureDataAdapter from a GeoTools SimpleFeatureType, adds it to the data store in the spatial index that was created in the previous section, and then uses the Writer to write some features:

Data in GeoWave can be queried by using the appropriate QueryBuilder implementation as described in the query builder documentation. The following is an example of using the VectorQueryBuilder to query feature data that lies within a specific bounding box:

Aggregation queries can be performed by using an AggregationQueryBuilder as described in the query builder documentation. The following is an example of performing a count aggregation on a vector type in the data store for features that lie in a given bounding box:

You can also create aggregations for any custom Aggregation implementation by using the aggregate function of the AggregationQueryBuilder.

Statistic queries can be performed by using an appropriate StatisticQueryBuilder. The following is an example of querying the bounding box statistic of a vector type in the data store:

The table below outlines the different project hierarchies within the GeoWave project

Both release and snapshot GeoWave artifacts are available on Maven Central.

GeoWave dependencies can be added through a Maven POM file with the following snippet (replacing ${geowave-artifact} with the desired artifact and ${geowave.version} with the desired version):

To use GeoWave snapshots the following repository should be added to the POM file:

The GeoWave Python bindings, also known as pygw provides a subset of GeoWave’s API to Python applications through the use of a Py4J Java Gateway. The Python bindings documentation has more information about the specifics of this API.

Using Jace, we are able to create JNI proxy classes for GeoWave that can be used in C/C++ applications.

Boost is required when using the Jace bindings.