Thunderhead Explorer

Josh Wills ( @Josh_wills ) introduced me to Apache Crunch  which is now a top-level project within the Apache Foundation. Crunch simplifies the coding of data aggregation in MapReduce. Here is a proof-of-concept project that spatially enables a crunch pipeline with a Point-In-Polygon function from a very large set of static point data with a small set of dynamic polygons. Crunch has simplified so much so the process, that is came down to a one line syntax: final PTable<Long, Long> counts = pipeline. readTextFile(args[0]). parallelDo(new PointInPolygon(), Writables.longs()). count(); Crunch's strength is in processing BigData that cannot be stored in the "traditional means", such a time series and graphs. Will be interesting to perform some kind to spatial and temporal analysis with it in a followup post. Like usual, all the source code can be found here .

In the previous post  we launched a CDH cluster on EC2 in under 5 minutes. In this post, we will use that cluster to perform geo spatial analytics in the form of MapReduce and visualize the result in ArcMap . See, ArcMap is one of the desktop tools that a GeoData Scientist will use when working and visualizing spatial data.  The use case in my mind is something like the following:  Point data is streamed through, for example GeoEventProcessor  into Amazon S3. The user has a set polygons in ArcGIS that needs to be spatially joined with that big data point content. The result of the big data join is linked back to the polygon set for symbol classification and visualization. After editing the polygons in ArcMap, the user exports the feature class into HDFS using the ExportToHDFSTool. Using the new  Esri Geometry API for Java , a MapReduceJob is written as a GeoProcessing extension, in such that it can be directly executed from within ArcMap. The result of the job is covered directly b...

well....after you get all the necessary software, certificates and... setup everything correctly :-) Update : Regarding the above comment - you can download a zip file containing all the necessary jars and the toolbox so like that you do not have to package the project from scratch. The idea here is that I would like an ArcGIS user to just push a button from within ArcMap  and have a Cloudera based Hadoop cluster started on Amazon EC2 . From there on, a user can edit features in ArcMap that can be exported into that cluster to be used as an input to a MapReduce job. The output of the MapReduce job is imported back into ArcMap for further analysis. This combination of SmallData (GeoDatabase) and BigData (Hadoop) is a great fusion in a geo-data scientist arsenal. When done with the analysis, the user again will push a button and destroys the cluster, thus paying for what he/she used while having access to elastic resources. The following is a sequence of prerequisite steps that you...