Thunderhead Explorer

This is a set of experiments with  HBase . From within ArcMap , you can create an HTable and export the content of a feature class into that table. The HTable is mapped into a Hive table to perform SQL Like queries on it and of course the Spatial Framework UDFs can be applied for spatial operations. If Hive is not fast enough for you, try Impala . Finally, by taking advantage of the RowKey  design, a spatial index is derived using geohash  to be later used in a MapReduce big spatial joins job. Like usual all the source code can be found here .

  In the GIS tools for Hadoop , we store and retrieve feature classes in Esri JSON or GeoJSON formats to and from HDFS. This post is about a set of experiments in different storage and serialization technics of feature classes. The two environments I evaluated are Apache Avro and Parquet . All the source code can be found here . Experiment I - Simple Feature Class Schema Despite that Avro has the capability of dynamic typing and being old fashion, I wanted to generate a set of POJOs for a simple feature class schema.  In my mind, a feature has a geometry and a set of attributes.   A geometry has a spatial reference and can be either a point, line or a polygon.  A point is made up of one coordinate. A coordinate is simply made up of an x and y value. Lines are made up of paths, where a path is a set of coordinates. Polygons are made of rings, where a ring is a set of coordinates. Based on the Avro specification , I wrote two schemas; one for a generic feature and one for a point f...