GeoWave EMR Quickstart Guide

This guide assumes that GeoWave has already been installed and is available on the command-line. See the Installation Guide for help with the installation process.

In order to keep things organized, create a directory on your system that can be used throughout the guide. The guide will refer to this directory as the working directory.

We will be using data from the GDELT Project in this guide. For more information about the GDELT Project please visit their website here.

Download one or more ZIP files from the GDELT Event Repository into a new gdelt_data folder in the working directory. The examples in this guide will use all of the data from February 2016 (201602 Prefix).

Later in the guide, we will be visualizing some data using GeoServer. For this, we will be using some styles that have been created for the demo.

Download the following styles to your working directory:

When finished, you should have a directory structure similar to the one below.

After all the data and styles have been downloaded, we can continue.

Depending on which key/value store that was configured in the EMR setup, execute the appropriate command to add the store to the GeoWave configuration, replacing $HOSTNAME with the Master public DNS of the EMR cluster:

This command adds a connection to the key/value store on EMR under the name gdelt for use in future commands. It configures the connection to put all data for this named store under the geowave.gdelt namespace.

Before ingesting any data, we need to create an index that describes how the data will be stored in the key/value store. For this example we will create a simple spatial index.

This command adds a spatial index to the gdelt data store with an index name of gdelt-spatial, which will be used to reference this index in future commands. It configured the index to use a round robin partitioning strategy with 32 partitions.

GeoWave has many commands that facilitate ingesting data into a GeoWave data store. For this example, we want to ingest GDELT data from the local file system, so we will use the ingest localToGW command. We will use a bounding box that roughly surrounds Germany to limit the amount of data ingested for the example.

This command specifies the input format as GDELT using the -f option, filters the input data using a CQL bounding box filter, and specifies the input directory for all of the files. Finally, we tell GeoWave to ingest the data to the gdelt-spatial index in the gdelt data store. GeoWave creates an adapter for the new data with the type name gdeltevent, which we can use to refer to this data in other commands. The ingest should take about 3-5 minutes.

Now that the data has been ingested, we can make queries against it. The GeoWave programmatic API provides a large variety of options for issuing queries, but for the purposes of this guide, we will use the query language support that is available for vector data. This query language provides a simple way to perform some of the most common types of queries using a well-known syntax. To demonstrate this, perform the following query:

This command tells GeoWave to select all attributes from the gdeltevent type in the gdelt data store, but limits the output to 10 features. After running this command, you should get a result that is similar to the following:

We can see right away that these results are tagged with the country code EZ which falls under Czech Republic. Since our area of interest is around Germany, perhaps we want to only see events that are tagged with the GM country code. We can do this by adding a WHERE clause to the query.

Now the results show only events that have the GM country code.

If we wanted to see how many events belong to to the GM country code, we can perform an aggregation query.

We can also perform multiple aggregations on the same data in a single query. The following query counts the number of entries that have set actor1Name and how many have set actor2Name.

We can also do bounding box aggregations. For example, if we wanted to see the bounding box of all the data that has HUNGARY set as the actor1Name, we could do the following:

For more information about queries, see the queries section of the User Guide.

We can also perform analytics on data that has been ingested into GeoWave. In this example, we will perform the Kernel Density Estimation (KDE) analytic.

This command tells GeoWave to perform a Kernel Density Estimation on the gdeltevent type. It specifies that the KDE should be run at zoom levels 5-26 and that the new raster generated should be under the type name gdeltevent_kde. It also specifies that the minimum and maximum splits should be 32, which is the number of partitions that were created for the index. It then points the analytic to the HDFS and resource manager ports on the EMR cluster. Finally, it specifies the input and output data store as our gdelt store. It is possible to output the results of the KDE to a different data store, but for this demo, we will use the same one. The KDE can take 5-10 minutes to complete due to the size of the dataset.

Now that we have prepared our vector and KDE data, we can visualize it by using the GeoServer plugin. GeoWave provides an embedded GeoServer with the command-line tools.

The Landsat 8 extension for GeoWave utilizes GDAL (Geospatial Data Abstraction Library), an image processing library, to process raster data. In order to use GDAL, native libraries need to be installed on the system. More info on GDAL can be found here.

GeoWave provides a way to install GDAL libraries with the following command:

For this demo, we will be using two data stores. One will be used for vector data, and the other will be used for raster data. Again, replace $HOSTNAME with the Master public DNS of the EMR cluster:

These commands creates a store for the raster data, and then copies that store configuration, changing only the namespace for the vector data store. The result is that the data for both stores will be on the same key/value store, but under different namespaces, so GeoWave will treat them as separate data stores.

Before ingesting our raster data, we will add a spatial index to both of the data stores.

This is similar to the command we used to add an index in the vector demo, but we have added an additional option to specify the Coordinate Reference System (CRS) of the data. Geospatial data often uses a CRS that is tailored to the area of interest. This can be a useful option if you want to use a CRS other than the default. After these commands have been executed, we will have spatial indices named spatial-idx on both data stores.

We can now see what Landsat 8 data is available for our area of interest.

This command tells GeoWave to analyze the B8 band of Landsat raster data over a bounding box that roughly surrounds Berlin, Germany. It prints out aggregate statistics for the area of interest, including the average cloud cover, date range, number of scenes, and the size of the data. Data for this operation is written to the landsat directory (specified by the -ws option), which can be used by the ingest step.

Now that we have analyzed the available data, we are ready to ingest it into our data stores.

There is a lot to this command, but you’ll see that it’s quite similar to the analyze command, but with some additional options. The --crop option causes the raster data to be cropped to our CQL bounding box. The --vectorstore landsatvector option specifies the data store to put the vector data (scene and band information). The --pyramid option tells GeoWave to create an image pyramid for the raster, this is used for more efficient rendering at different zoom levels. The --coverage berlin_mosaic option tells GeoWave to use berlin_mosaic as the type name for the raster data. Finally, we specify the output data store for the raster, and the index to store it on.

We will once again use GeoServer to visualize our ingested data.