Creating Spatial Crunch Pipelines

Josh Wills (@Josh_wills) introduced me to Apache Crunch which is now a top-level project within the Apache Foundation. Crunch simplifies the coding of data aggregation in MapReduce.

Here is a proof-of-concept project that spatially enables a crunch pipeline with a Point-In-Polygon function from a very large set of static point data with a small set of dynamic polygons.

Crunch has simplified so much so the process, that is came down to a one line syntax:

final PTable<Long, Long> counts = pipeline. readTextFile(args[0]). parallelDo(new PointInPolygon(), Writables.longs()). count();

Crunch's strength is in processing BigData that cannot be stored in the "traditional means", such a time series and graphs. Will be interesting to perform some kind to spatial and temporal analysis with it in a followup post.

Like usual, all the source code can be found here.

Comments

Post a Comment

Popular posts from this blog

How to use Esri Flex API on Android and iPhone

Image
Update (Jun 2011) : For more updated info, check out my ADC article here . At MAX2010 (Awesome BTW) Adobe introduced Flex SDK Hero and Flash Builder Burrito which enables you to build and deploy Flex mobile applications on Android by default. In addition, using the revitalized Packager for iPhone , you can postprocess the swf into an ipa that you can deploy on an iPhone using iTunes. In this post, I will share with you the steps that I took to use the Flex API for ArcGIS to build a mobile mapping application. First and foremost - RTFM - here are some links that u _need_ to read specially for iPhone provisioning: http://opensource.adobe.com/wiki/display/flexsdk/Hero http://labs.adobe.com/technologies/flashbuilder_burrito/ http://labs.adobe.com/technologies/packagerforiphone/ The cool part of this whole process is now you have one code base to build upon and deploy on Android and iPhone and in the future whatever mobile platform Adobe decides to support. You can download ...
Read more

Weighted Map Feature Clustering With Attributes

Image
Was asked if the WeightedClusterer can cluster features not just by proximity but by proximity and attribute property value. The idea is that there should be at least one cluster per unique attribute property value. Let me explain further:  The above snapshot is a clustering of cities based on proximity and country - note that despite the proximity of Belgium and The Netherlands, two distinct clusters have been created one for BE and the other for NL - now for France - there is a cluster containing one feature over Paris, however due to the distance from Paris (at that scale) a cluster was created based on 3 cities in the south of France. Make sense ? In the implementation of the AttrClusterer class that extends the ESRIClusterer class, we override the clusterGraphics function. This function is invoked indirectly by the Flex framework during the update of the display list. In this function, we overlay a virtual grid on the map. This grid acts as a spatial index. Features that are ...
Read more

Augmented Reality, iPhone and ArcGIS Server

Image
I am a HUGE fan of Augmented Reality , and one of my favorite applications on the iPhone is Layar . In this post, I will "walk you" through how to setup a simple ArcGIS Layar Point Of Interests (POI) layer and how to expose it through the iPhone application. In addition, I created a small Flex application that will enable you to add POIs to that feature class through a python geo-processor task. BTW - this is heavily geared to using ESRI desktop and server tools and I'm assuming that you are familiar with these tools. First, request a developer key from http://dev.layar.com/ . This might take a couple of hours to get to you. So, in the meantime, let's get started. Using ArcCatalog, create a new point feature class called 'Layar' in a geodatabase with the following fields: Title (text), Line2 (Text), Line3 (Text), Line4 (Text), Attribution (Text), Type (Integer), ImageUrl (Text:512). Import that layer into a map document using ArcMap. Save and publish the ...
Read more