Thunderhead Explorer

In the last post, we walked through how to setup Elasticsearch in a Docker container and how to bulk load the content of an ArcGIS feature class into ES, in such that it can be spatially searchable from an ArcPy based tool. There was something nagging me about my mac development environment, as I was using docker in VirtualBox and ArcGIS Desktop on Windows in WMWare Fusion . I wish I had one unified virtualized environment. Well, while at MesosCon in Seattle, I stopped by the VMWare booth and the folks there told me about a new project named Photon™ . It is "a minimal Linux container host. It is designed to have a small footprint and boot extremely quickly on VMware platforms. Photon™ is intended to invite collaboration around running containerized applications in a virtualized environment.” - That was exactly what I needed, and docker is built into it ! See, what also got me excited, was the fact that in a couple of weeks, I will be visiting a very forward thinking...

I really like Elasticsearch because it natively supports geo spatial types and queries. I just added to gitbub a ArcPy based toolbox to bulk load the content of a feature class into an ES index/type. The toolbox contains yet another tool as a proof-of-concept to spatially query the loaded document.

I’m always handed a huge set of CSV records with lat/lon coordinates, and the task at hand is to spatially join these records with a huge set of feature polygons where the output is a geoenhancement of the orignal points with the intersecting polygon’s attributes. An example is a set of retailer customer locations that need to be spatially intersected with demographic polygons for targeted advertisement (Sorry to send you all more junk mail :-). This is a reference implementation, where both the points data and the polygon data are stored in raw text TSV format and the polygon geometries are in WKT format. Not the most efficient format, but at least the input is splittable for massive parallelization. The feature class polygons can be converted to WKT using this ArcPy tool. This Spark based job can be executed in local mode, or better in this docker container. One of these days will have to re-implement the reading of the polygons from a binary source such as shapefiles or file g...