Thunderhead Explorer

Artificial Intelligence / Machine Learning implementations have been part of Esri software in ArcGIS for a very long time. Geographic Weighted Regression (GWR) and Hot Spot Analysis are Machine Learning algorithms. ArcGIS users have been utilizing supervised and   unsupervised learning like clustering  to solve a myriad of problems and gain geospatial insight from their data.  We just did not call these learning algorithms AI/ML back then.  Not until the recent popularity of DeepLearning that finally blossomed from the AI Winter  and made AI a household name. And guess what? Esri software has now DeepLearning implementations ! It is important to understand the difference between AI, ML, and DL.  I came across this very insightful article that analogizes the relationship to Russian dolls . Typically, collected data is very "dirty", and a lot of cleaning has to performed on it before processing it through a Machine Learning algorithm.  The bigger ...

Yet another post in the continuing saga of the usage of Apache Spark from a Jupyter notebook within ArcGIS Pro . In the previous posts , the execution was always within the ArcGIS Pro environment on a single machine, albeit taking advantage of all the cores of that machine.  Here, we take a different angle, the execution is performed on a remote cluster of machines in the cloud. So, we author the notebook locally, but we execute it remotely. In this notebook, we demonstrate the spatial binning of AIS broadcast points on a Databricks cluster on Azure . In addition, to colocate the data storage with the execution engine for performance purposes, we converted the local feature class of the AIS broadcast points to a parquet file and placed it in the Databricks distributed file system . More to come :-)

Yet another continuation post regarding Pro, Notebook, and Spark :-). In this notebook, we will demonstrate  a parallel, distributed, share-nothing spatial join between a relatively large dataset and a small dataset. In this case, virtual gates are defined at various locations in a port, and the outcome is an account of the number of crossings of these gates by ships using their AIS target positions. Note that the join is to a "small" spatial dataset that we can: Broadcast  to all the spark workers. Brutly traverse it on each worker, as it is cheaper and faster to do so that spatially index it. The following are sample gates: And the following is a sample processed output: More to come...

This is a continuation of the last post regarding ArcGIS Pro, Jupyter Notebook, and Spark. And, this is a rehash of an older post in a more "modern" way. Micropathing is the construction of a target's path from a limited set of a consecutive sequence of target points. Typically, the sequence is time-based, and the collection is limited to 2 or 3 target points.   The following is an illustration of 2 micropaths derived from 3 target points: Micropathing is different than path reconstruction, in such that the latter produced one polyline for the path of a target. Path reconstruction losses insightful in-path behavior, as a large number of attributes cannot be associated with the path parts. Some can argue that the points along the path can be enriched with these attributes. However, with the current implementations of Point objects, we are limited to only the extra M  and Z to the necessary X and Y . You can also join the PathID and M to a lookup table and gain back th...