Calculating weighted variance using Spark

Spark provides a stat() function on a DoubleRDD to calculate in a robust way the count, mean and standard deviation of the double values. The inputs are unweighted (or all have a weight of 1), and I ran into a situation where I needed to perform the statistics on weighted values. The use case is to find the density area of locations with associated number of incidents. Here is the result on a map:
So, after re-reading the "Weighted Incremental Algorithm" section on wikipedia, I ripped the original StatCounter code and and implemented a WeightedStatCounter class that calculates the statistics for an RDD of WeightedValues. The above use case is based on the Standard Distance GeoProcessing function and generates a WKT Polygon that is converted to a Feature through an ArcPy Toolbox.

Works like a charm, and like usual, all the source code can be found here.

Comments

Post a Comment

Popular posts from this blog

How to use Esri Flex API on Android and iPhone

Image
Update (Jun 2011) : For more updated info, check out my ADC article here . At MAX2010 (Awesome BTW) Adobe introduced Flex SDK Hero and Flash Builder Burrito which enables you to build and deploy Flex mobile applications on Android by default. In addition, using the revitalized Packager for iPhone , you can postprocess the swf into an ipa that you can deploy on an iPhone using iTunes. In this post, I will share with you the steps that I took to use the Flex API for ArcGIS to build a mobile mapping application. First and foremost - RTFM - here are some links that u _need_ to read specially for iPhone provisioning: http://opensource.adobe.com/wiki/display/flexsdk/Hero http://labs.adobe.com/technologies/flashbuilder_burrito/ http://labs.adobe.com/technologies/packagerforiphone/ The cool part of this whole process is now you have one code base to build upon and deploy on Android and iPhone and in the future whatever mobile platform Adobe decides to support. You can download ...
Read more

Weighted Map Feature Clustering With Attributes

Image
Was asked if the WeightedClusterer can cluster features not just by proximity but by proximity and attribute property value. The idea is that there should be at least one cluster per unique attribute property value. Let me explain further:  The above snapshot is a clustering of cities based on proximity and country - note that despite the proximity of Belgium and The Netherlands, two distinct clusters have been created one for BE and the other for NL - now for France - there is a cluster containing one feature over Paris, however due to the distance from Paris (at that scale) a cluster was created based on 3 cities in the south of France. Make sense ? In the implementation of the AttrClusterer class that extends the ESRIClusterer class, we override the clusterGraphics function. This function is invoked indirectly by the Flex framework during the update of the display list. In this function, we overlay a virtual grid on the map. This grid acts as a spatial index. Features that are ...
Read more

Augmented Reality, iPhone and ArcGIS Server

Image
I am a HUGE fan of Augmented Reality , and one of my favorite applications on the iPhone is Layar . In this post, I will "walk you" through how to setup a simple ArcGIS Layar Point Of Interests (POI) layer and how to expose it through the iPhone application. In addition, I created a small Flex application that will enable you to add POIs to that feature class through a python geo-processor task. BTW - this is heavily geared to using ESRI desktop and server tools and I'm assuming that you are familiar with these tools. First, request a developer key from http://dev.layar.com/ . This might take a couple of hours to get to you. So, in the meantime, let's get started. Using ArcCatalog, create a new point feature class called 'Layar' in a geodatabase with the following fields: Title (text), Line2 (Text), Line3 (Text), Line4 (Text), Attribution (Text), Type (Integer), ImageUrl (Text:512). Import that layer into a map document using ArcMap. Save and publish the ...
Read more