Thunderhead Explorer

This is a set of experiments with  HBase . From within ArcMap , you can create an HTable and export the content of a feature class into that table. The HTable is mapped into a Hive table to perform SQL Like queries on it and of course the Spatial Framework UDFs can be applied for spatial operations. If Hive is not fast enough for you, try Impala . Finally, by taking advantage of the RowKey  design, a spatial index is derived using geohash  to be later used in a MapReduce big spatial joins job. Like usual all the source code can be found here .

  In the GIS tools for Hadoop , we store and retrieve feature classes in Esri JSON or GeoJSON formats to and from HDFS. This post is about a set of experiments in different storage and serialization technics of feature classes. The two environments I evaluated are Apache Avro and Parquet . All the source code can be found here . Experiment I - Simple Feature Class Schema Despite that Avro has the capability of dynamic typing and being old fashion, I wanted to generate a set of POJOs for a simple feature class schema.  In my mind, a feature has a geometry and a set of attributes.   A geometry has a spatial reference and can be either a point, line or a polygon.  A point is made up of one coordinate. A coordinate is simply made up of an x and y value. Lines are made up of paths, where a path is a set of coordinates. Polygons are made of rings, where a ring is a set of coordinates. Based on the Avro specification , I wrote two schemas; one for a generic feature and one for a point f...

So in this short post I am integrating 4 things that I love the most - GIS, Hadoop, BigMemory and Minecraft - The idea is the following, I would like to visualize in Minecraft the result of Kernel Density calculation performed by Hadoop.  The job output is placed into distributed memory rather than back into HDFS in such that a Minecraft server can read that memory and populate a world that I have teleported into. Here is a short video that showcases the last steps after I started the BigMemory server and ran the Hadoop Job. Like usual all the source code is available here .

At this year's User Conference , I presented in the Esri Labs booth "The Gamification of GIS", where I showcased how a Minecraft  player can teleport and interact with GIS worlds. It was a simple proof of concept, but I believe the ramification can be significant, especially after listening to Amber Case talking about STEM in the plenary. My boys and their friends can play  Minecraft  for hours. They build collaboratively amazing online worlds and pixel art . And despite the "blockiness" of the worlds and the art, the game is pretty immersive. One day my son tells me "Hey dad, I think I can build one of your worlds in Minecraft". What he meant about my world is a GIS world that I render using ArcMap . So I started thinking about this and wondered if I too can do this, but programmatically. A bit of googling, and found out that I can program the Minecraft server using Java .  My boys play Minecraft on their XBox and have a desktop version too. T...

Josh Wills ( @Josh_wills ) introduced me to Apache Crunch  which is now a top-level project within the Apache Foundation. Crunch simplifies the coding of data aggregation in MapReduce. Here is a proof-of-concept project that spatially enables a crunch pipeline with a Point-In-Polygon function from a very large set of static point data with a small set of dynamic polygons. Crunch has simplified so much so the process, that is came down to a one line syntax: final PTable<Long, Long> counts = pipeline. readTextFile(args[0]). parallelDo(new PointInPolygon(), Writables.longs()). count(); Crunch's strength is in processing BigData that cannot be stored in the "traditional means", such a time series and graphs. Will be interesting to perform some kind to spatial and temporal analysis with it in a followup post. Like usual, all the source code can be found here .

In the previous post  we launched a CDH cluster on EC2 in under 5 minutes. In this post, we will use that cluster to perform geo spatial analytics in the form of MapReduce and visualize the result in ArcMap . See, ArcMap is one of the desktop tools that a GeoData Scientist will use when working and visualizing spatial data.  The use case in my mind is something like the following:  Point data is streamed through, for example GeoEventProcessor  into Amazon S3. The user has a set polygons in ArcGIS that needs to be spatially joined with that big data point content. The result of the big data join is linked back to the polygon set for symbol classification and visualization. After editing the polygons in ArcMap, the user exports the feature class into HDFS using the ExportToHDFSTool. Using the new  Esri Geometry API for Java , a MapReduceJob is written as a GeoProcessing extension, in such that it can be directly executed from within ArcMap. The result of the job is covered directly b...

well....after you get all the necessary software, certificates and... setup everything correctly :-) Update : Regarding the above comment - you can download a zip file containing all the necessary jars and the toolbox so like that you do not have to package the project from scratch. The idea here is that I would like an ArcGIS user to just push a button from within ArcMap  and have a Cloudera based Hadoop cluster started on Amazon EC2 . From there on, a user can edit features in ArcMap that can be exported into that cluster to be used as an input to a MapReduce job. The output of the MapReduce job is imported back into ArcMap for further analysis. This combination of SmallData (GeoDatabase) and BigData (Hadoop) is a great fusion in a geo-data scientist arsenal. When done with the analysis, the user again will push a button and destroys the cluster, thus paying for what he/she used while having access to elastic resources. The following is a sequence of prerequisite steps that you...