Thunderhead Explorer

There has been a lot of research on performing spatial joins using Hadoop MapReduce on BigData, specially when both sets are very big. A notable one is the Spatial Hadoop  project at the University of Minnesota. This post is derived from that body of work and uses the Esri Geometry API  in the GIS Tools for Hadoop project to perform the spatial operations in the map and reduce phases. Lexicographical join of native types (numerical,textural) in BigData can be performed in the map phase or in the reduce phase depending on the size of the input data and how much memory the mapper or reducer has access to. Actually, Hadoop provides a join framework package as this is a common pattern. The same can be applied for spatially joining two big data sets. It can be performed in the mapper phase or the reducer phase depending on the size of the data and how much memory each phase has access to. Let's start with a map phase spatial join. Let's say you have a billion point records and ...

In this post, I will show how to log very large amount of web requests to a BigData storage for traffic analysis. The source code for the project is on github . We will rely on the logging library log4j and the associated  Flume NG  appender  implementation . For storage, we will place the log information into a set of  HDFS  files or into an  HBase  table. The HDFS files will be mapped into a Hive table as partitions for query and aggregation. In this demonstration, all web requests are handled by a very simple web application  servlet that is mapped to an access url: In practice, all logic is handled by the servlet and the logging of the request information will be handled by a servlet filter that uses the log4j logging API. In this demonstration the path info will define the log level and the query string will define the log message content. Log4j is configured using the resource log4j.properties to use a Flume NG appender: The ...

I am now a Cloudera Certified Specialist in Apache HBase . Woohoo !!!

What is GeoEnrichment? An example would best describe it. Given a big set of customer location records, I would like each location to be GeoEnriched with the average income of the zip code where that location falls into and with the number of people between the age of 25 and 30 that live in that zip code. Before GeoEnrichment: CustId,Lat,Lon After GeoEnrichment: CustId,Lat,Lon, AverageIncome,Age25To30 Of course the key to this whole thing is the spatial reference data :-) and there are a lot of search options, such as Point-In-Polygon, Nearest Neighbor and enrichment based on a Drive Time Polygon from each location. I've implemented two search methods: Point-In-Polygon method Nearest Neighbor Weighted method The Point-In-Polygon (PiP) method is fairly simple. Given a point, find the polygon it falls into and pull from the polygon feature the selected attributes and add them to the original point. The Nearest Neighbor Weighted (NNW) method finds all the reference...

This is a set of experiments with  HBase . From within ArcMap , you can create an HTable and export the content of a feature class into that table. The HTable is mapped into a Hive table to perform SQL Like queries on it and of course the Spatial Framework UDFs can be applied for spatial operations. If Hive is not fast enough for you, try Impala . Finally, by taking advantage of the RowKey  design, a spatial index is derived using geohash  to be later used in a MapReduce big spatial joins job. Like usual all the source code can be found here .

  In the GIS tools for Hadoop , we store and retrieve feature classes in Esri JSON or GeoJSON formats to and from HDFS. This post is about a set of experiments in different storage and serialization technics of feature classes. The two environments I evaluated are Apache Avro and Parquet . All the source code can be found here . Experiment I - Simple Feature Class Schema Despite that Avro has the capability of dynamic typing and being old fashion, I wanted to generate a set of POJOs for a simple feature class schema.  In my mind, a feature has a geometry and a set of attributes.   A geometry has a spatial reference and can be either a point, line or a polygon.  A point is made up of one coordinate. A coordinate is simply made up of an x and y value. Lines are made up of paths, where a path is a set of coordinates. Polygons are made of rings, where a ring is a set of coordinates. Based on the Avro specification , I wrote two schemas; one for a generic feature and one for a point f...

So in this short post I am integrating 4 things that I love the most - GIS, Hadoop, BigMemory and Minecraft - The idea is the following, I would like to visualize in Minecraft the result of Kernel Density calculation performed by Hadoop.  The job output is placed into distributed memory rather than back into HDFS in such that a Minecraft server can read that memory and populate a world that I have teleported into. Here is a short video that showcases the last steps after I started the BigMemory server and ran the Hadoop Job. Like usual all the source code is available here .

At this year's User Conference , I presented in the Esri Labs booth "The Gamification of GIS", where I showcased how a Minecraft  player can teleport and interact with GIS worlds. It was a simple proof of concept, but I believe the ramification can be significant, especially after listening to Amber Case talking about STEM in the plenary. My boys and their friends can play  Minecraft  for hours. They build collaboratively amazing online worlds and pixel art . And despite the "blockiness" of the worlds and the art, the game is pretty immersive. One day my son tells me "Hey dad, I think I can build one of your worlds in Minecraft". What he meant about my world is a GIS world that I render using ArcMap . So I started thinking about this and wondered if I too can do this, but programmatically. A bit of googling, and found out that I can program the Minecraft server using Java .  My boys play Minecraft on their XBox and have a desktop version too. T...

Josh Wills ( @Josh_wills ) introduced me to Apache Crunch  which is now a top-level project within the Apache Foundation. Crunch simplifies the coding of data aggregation in MapReduce. Here is a proof-of-concept project that spatially enables a crunch pipeline with a Point-In-Polygon function from a very large set of static point data with a small set of dynamic polygons. Crunch has simplified so much so the process, that is came down to a one line syntax: final PTable<Long, Long> counts = pipeline. readTextFile(args[0]). parallelDo(new PointInPolygon(), Writables.longs()). count(); Crunch's strength is in processing BigData that cannot be stored in the "traditional means", such a time series and graphs. Will be interesting to perform some kind to spatial and temporal analysis with it in a followup post. Like usual, all the source code can be found here .

In the previous post  we launched a CDH cluster on EC2 in under 5 minutes. In this post, we will use that cluster to perform geo spatial analytics in the form of MapReduce and visualize the result in ArcMap . See, ArcMap is one of the desktop tools that a GeoData Scientist will use when working and visualizing spatial data.  The use case in my mind is something like the following:  Point data is streamed through, for example GeoEventProcessor  into Amazon S3. The user has a set polygons in ArcGIS that needs to be spatially joined with that big data point content. The result of the big data join is linked back to the polygon set for symbol classification and visualization. After editing the polygons in ArcMap, the user exports the feature class into HDFS using the ExportToHDFSTool. Using the new  Esri Geometry API for Java , a MapReduceJob is written as a GeoProcessing extension, in such that it can be directly executed from within ArcMap. The result of the job is covered directly b...

well....after you get all the necessary software, certificates and... setup everything correctly :-) Update : Regarding the above comment - you can download a zip file containing all the necessary jars and the toolbox so like that you do not have to package the project from scratch. The idea here is that I would like an ArcGIS user to just push a button from within ArcMap  and have a Cloudera based Hadoop cluster started on Amazon EC2 . From there on, a user can edit features in ArcMap that can be exported into that cluster to be used as an input to a MapReduce job. The output of the MapReduce job is imported back into ArcMap for further analysis. This combination of SmallData (GeoDatabase) and BigData (Hadoop) is a great fusion in a geo-data scientist arsenal. When done with the analysis, the user again will push a button and destroys the cluster, thus paying for what he/she used while having access to elastic resources. The following is a sequence of prerequisite steps that you...

I was asked to put a Proof of Concept implementation of a very fast interactive dynamic density map generation on 11 million records for a webmap application , where a user can specify dynamically a query definition (a where clause in SQL terms), a ramp color and the service implementation will return back a density representation of the records on a map. This is typically done via a GeoProcessing task where the data is queried and stored into an intermediate FeatureClass that is further processed by a Kernel Density that produced a raster layer that is finally visualized. As you can tell, this is not interactive nor relatively fast. Since the traditional means of retrieving the data from a relational database is not fast enough and 11 million records is not such a big set after all, I decided to put the whole thing in memory. BTW, this is a meme that has been trending for while now, and the most vocal about it is SAP HANA . I decided to use Terracotta's BigMemory to hold t...

At this year's DevSummit , we announced the GIS Tools for Hadoop  project. Included in that project is a low level geometry java API which enables spatial operations in MapReduce jobs or the construction of higher level functions such as Hive User Defined Functions . However, this geometry library is not restricted to Hadoop MapReduce. It is used in Geo Event Processor , and can be used in Storm bolts or other parallel workflows. One such parallel workflow processing engine is Pervasive DataRush that I demoed at the DevSummit. Using the KNIME visual workflow , I was able to process 90 million records (BTW, small in the BigData world) in Hadoop File System, for heatmap visualization in ArcMap. A DataRush workflow engine can run on a local machine or remotely on a cluster of machines and is fully extensible with custom operators.  An operator is a node in a workflow graph whose input and output can be linked to other operators. So, I wrote my own spatial operators tha...

The Raspberry Pi is a very neat little machine - I bootstrapped it with Soft-Float Debian "Wheezy"  so I can run Java Embedded on it - I download and install  ejre-7u10-fcs-b18-linux-arm-vfp-client_headless-28_nov_2012.tar , and now can run java programs on the Pi. Wrote the ubiquitous "Hello World" to make sure it works (I should not have doubted the write once run anywhere motto) and jumped directly to running a Spring container with beans. So, the next thing to do is to run the Hadoop client from my previous post on it.  Now remember, there is not a lot of RAM on the Model B Rev 2 Pi (512MB) and the post-action runs on the client machine, so I reduced the cell size and the search radius, and....it worked like a charm. Now, why did I do that ? because... I can :-) Next, is to come up with something to do with the generated data.

The storage of BigData has been democratized by Hadoop. And with that, rather than bringing the data to the program for analysis, we send the program to the data. That twist comes with a challenge to take existing serial algorithms and parallelizing them. This post is about such a process. I want to perform a Kernel Density analysis on billions of records loaded into HDFS. The data is fairly simple; a set of records separated by carriage return, and the fields in each record are tab separated. The fields' values contain a latitude, a longitude and a weight. If you do not know how Kernel Density works, check out this ArcGIS reference. To take advantage of the distributed HDFS datanodes, I have to transform the traditional state-full sequential implementation into a model that supports stateless parallelism. MapReduce is such a model, and Hadoop can distribute a MapReduce implementation to read data from the "local" HDFS datanodes and reduces back the result into HD...