Spark SQL DBF Library

Happy new year all…It’s been a while. I was crazy busy from May till mid December of last year implementing BigData  geospatial solutions at client sites all over the world. Was in Japan a couple of times, Singapore, Malaysia, UK, and do not recall the times I was in Redlands, Texas and DC.  In addition, I’ve been investing heavily in Spark and Scala. Do not recall the last time I implemented a Hadoop MapReduce job !

One of the resolutions for the new year (in addition to the usual eating right, exercising more and the never-off-the-bucket-list biking Mt Ventoux) is to blog more. One post per month as a minimum.

So…to kick to year right, I’ve implemented a library to query DBF files using Spark SQL. With the advent of Spark 1.2, a custom relation (table) can be defined as a SchemaRDD.  A sample implementation is demonstrated by Databrick’s spark-avro, as Avro files have embedded schema and data so it is relatively easy to convert that to a SchemaRDD. We, in the geo community have such a “old” format that encapsulates schema and data; the DBF format. Using the Shapefile project, I was able to create an RDD using the spark context Hadoop file API and the implementation of a DBFInputFormat. Then using the DBFHeader fields information, each record was mapped onto a Row to be processed by SparkSQL.  This is mostly work in progress and is far from been optimized, but it works !


Like usual, all the source code can be downloaded from here. Happy new year all.

Comments

Post a Comment

Popular posts from this blog

How to use Esri Flex API on Android and iPhone

Image
Update (Jun 2011) : For more updated info, check out my ADC article here . At MAX2010 (Awesome BTW) Adobe introduced Flex SDK Hero and Flash Builder Burrito which enables you to build and deploy Flex mobile applications on Android by default. In addition, using the revitalized Packager for iPhone , you can postprocess the swf into an ipa that you can deploy on an iPhone using iTunes. In this post, I will share with you the steps that I took to use the Flex API for ArcGIS to build a mobile mapping application. First and foremost - RTFM - here are some links that u _need_ to read specially for iPhone provisioning: http://opensource.adobe.com/wiki/display/flexsdk/Hero http://labs.adobe.com/technologies/flashbuilder_burrito/ http://labs.adobe.com/technologies/packagerforiphone/ The cool part of this whole process is now you have one code base to build upon and deploy on Android and iPhone and in the future whatever mobile platform Adobe decides to support. You can download ...
Read more

Augmented Reality, iPhone and ArcGIS Server

Image
I am a HUGE fan of Augmented Reality , and one of my favorite applications on the iPhone is Layar . In this post, I will "walk you" through how to setup a simple ArcGIS Layar Point Of Interests (POI) layer and how to expose it through the iPhone application. In addition, I created a small Flex application that will enable you to add POIs to that feature class through a python geo-processor task. BTW - this is heavily geared to using ESRI desktop and server tools and I'm assuming that you are familiar with these tools. First, request a developer key from http://dev.layar.com/ . This might take a couple of hours to get to you. So, in the meantime, let's get started. Using ArcCatalog, create a new point feature class called 'Layar' in a geodatabase with the following fields: Title (text), Line2 (Text), Line3 (Text), Line4 (Text), Attribution (Text), Type (Integer), ImageUrl (Text:512). Import that layer into a map document using ArcMap. Save and publish the ...
Read more

Weighted Map Feature Clustering With Attributes

Image
Was asked if the WeightedClusterer can cluster features not just by proximity but by proximity and attribute property value. The idea is that there should be at least one cluster per unique attribute property value. Let me explain further:  The above snapshot is a clustering of cities based on proximity and country - note that despite the proximity of Belgium and The Netherlands, two distinct clusters have been created one for BE and the other for NL - now for France - there is a cluster containing one feature over Paris, however due to the distance from Paris (at that scale) a cluster was created based on 3 cities in the south of France. Make sense ? In the implementation of the AttrClusterer class that extends the ESRIClusterer class, we override the clusterGraphics function. This function is invoked indirectly by the Flex framework during the update of the display list. In this function, we overlay a virtual grid on the map. This grid acts as a spatial index. Features that are ...
Read more