Thunderhead Explorer

Continuing the GenAI-with-a-GeoSpatial-twist thread I started a while back. Back then, I had a language model reason about geospatial logic. This time, I wanted it to actually do the work—on my open project, inside my running ArcGIS Pro session, while I watch. The result is ProCowork , an experimental native ArcGIS Pro add-in that embeds the Claude Code engine in a dockable chat panel. I can ask it to list layers, add and calculate fields, select features, run a buffer, or write a more specialized ArcPy script. Claude generates the code, the add-in runs it against the live project, and the code and results come back into the same panel. For example, I can type: “List the layers in the current map.” “Add a DOUBLE field POP_DEN to Parcels and set it to POP / AREASQMI.” “Select parcels where POP_DEN > 5000 and zoom to them.” “Buffer Roads by 100 meters and add the result to the map.” The panel is useful, but the interesting part is ...

Yet another post in the continuing saga of the usage of Apache Spark from a Jupyter notebook within ArcGIS Pro . In the previous posts , the execution was always within the ArcGIS Pro environment on a single machine, albeit taking advantage of all the cores of that machine.  Here, we take a different angle, the execution is performed on a remote cluster of machines in the cloud. So, we author the notebook locally, but we execute it remotely. In this notebook, we demonstrate the spatial binning of AIS broadcast points on a Databricks cluster on Azure . In addition, to colocate the data storage with the execution engine for performance purposes, we converted the local feature class of the AIS broadcast points to a parquet file and placed it in the Databricks distributed file system . More to come :-)

Yet another continuation post regarding Pro, Notebook, and Spark :-). In this notebook, we will demonstrate  a parallel, distributed, share-nothing spatial join between a relatively large dataset and a small dataset. In this case, virtual gates are defined at various locations in a port, and the outcome is an account of the number of crossings of these gates by ships using their AIS target positions. Note that the join is to a "small" spatial dataset that we can: Broadcast  to all the spark workers. Brutly traverse it on each worker, as it is cheaper and faster to do so that spatially index it. The following are sample gates: And the following is a sample processed output: More to come...

This is a continuation of the last post regarding ArcGIS Pro, Jupyter Notebook, and Spark. And, this is a rehash of an older post in a more "modern" way. Micropathing is the construction of a target's path from a limited set of a consecutive sequence of target points. Typically, the sequence is time-based, and the collection is limited to 2 or 3 target points.   The following is an illustration of 2 micropaths derived from 3 target points: Micropathing is different than path reconstruction, in such that the latter produced one polyline for the path of a target. Path reconstruction losses insightful in-path behavior, as a large number of attributes cannot be associated with the path parts. Some can argue that the points along the path can be enriched with these attributes. However, with the current implementations of Point objects, we are limited to only the extra M  and Z to the necessary X and Y . You can also join the PathID and M to a lookup table and gain back th...

Been a while since I posted something, and thank you, faithful reader, for coming back :-) I'm intending to writing a series of posts on how to use Apache Spark and Machine Learning within a jupyter notebook within ArcGIS Pro.  Yes, you can now start a jupyter notebook instance in ArcGIS Pro to create an amazing data science and data exploration experience. Check out this link to see how to get started with a Jupyter notebook in Pro.  But...my favorite hidden "GeoGem", is that Pro comes with built-in Apache Spark, and y'all know how much I love Spark. People think that Spark is intended for only BigData analytics.  That is so far from the truth. What I love about it, is the frictionless movement of data and analysis locally or remotely and the language fusion.  In my case, I'm using Python, SQL, and Scala. The usage of Apache Spark in Pro was demonstrated in the publically shared Covid-19 Contact Tracing Application  and the Proximity Tracing Application . In...

Been wanting to post on this subject for quite some time (actually over a year) as associating a world coordinate with the proper nearby linear feature provides tremendous insight based on the fusion of their attributes. Moreover, doing that on a massive scale and quickly is even more imperative in today's BigData world, thus the usage of Apache Spark . I’ve posted a standalone implementation that relies on well-documented simple math and published methodology to perform searches on massive datasets in batch mode. What is exciting to me in writing this post was the viewing of the snap results in ArcGIS Pro . My lack of knowledge in extending ArcGIS Pro with downloadable Python modules contributed to the delay (and slight case of procrastination :-). However, with the help of a colleague, I was able to pip install modules that can be imported by my custom ArcPy based toolboxes without any errors. Also, since this is all based on BigData, well it has to be tested in a BigData e...