Thursday, March 22, 2012

Enter the Fifth Dimension; Sentiment Map Rendering

We have been plotting x,y,z and time for a while now. But this is all syntax; Tim Berners-Lee believes that the future of the web is semantics, and I could not agree more. You have a glimpse of it today when you use for example Apple’s Siri. In our GIS space, MetaCarta at one time had a great appliance/service that inferred location from the semantics of a sentence. For example, “Joe ate a Chicago style pizza in downtown Boston”, would return the latitude and longitude of Downtown Boston and would ignore “Chicago” as a location, Impressive! With the explosion of social media and the infusion of geo-locations as ‘fields’ in our records, we are reduced to plotting locations with cute popups. Boring! We humans are quite the emotional animals (some more that others, specially if these humans come from the Middle East :-) and these emotions are quite visible now within our tweets and facebook posts. So, what if I can plot these sentiments, there could be something to “see” in these maps. Enter LinguaSys, I met one of their senior scientists by accident when sheltering from a rain storm. We ended up talking about the weather, this and that and of course the question “What do you do ?” had to eventually come up. “I do sentiment analysis”, he replied. “Wow, that is exactly what I was researching before leaving”, I replied. “I am looking for an ‘engine’ that I can pass it, say a set of tweets, and it will return to me for each tweet a sentiment index.”. “Not sure about tweets”, he replied, “as we deal with entire documents, but I am sure we can adjust our engine to such a process”. The rain stopped, we exchanged contact info and parted on the promise that we will stay in touch. A couple of month later, a very good customer with receptive avant-garde ideas needed something ‘new’. I proposed the sentiment index mapping based on social media to highlight areas of interest. The following is a derivative of this work based on a totally different interest; TSA approval or disapproval tweets. Working with LinguaSys, I was handed a set of XML files where one contained the tweets and associated sentiment indexes, for example:
<text>RT @msnbc_travel: Good news for elderly fliers (75 and above): TSA announces pilot program the relaxes security procedures http:\/\/t.co\/WTaM9AgW</text>
<disapprovalFactor factor="-0.8" reason=“indicatorOfSatisfaction”/>
Note that the factor is negative to indicate a level of satisfaction. The internal factor ranges from -1 (totally satisfied) to 1 (totally pissed off :-). Easy to parse and to associate a range based renderer. The second file is the “location” of the tweeters. Notice that I put location between quotes, that is because some values were great, like an exact latitude/longitude. Others were like “Boston, MA” And some (and most) were like “Earth”, or “Best Location, NYC!” or my favorite “Look behind u… Boo!”. There was a “sense” of location in these that I think would have given MetaCarta a run for its money. This is where being a unix CLI geek with tools like awk, grep and sed come to the rescue to massage the data. I downloaded a cities.csv to cross reference a city/state name to a location and now I can plot on a map the locatable tweets. Using the latest built-in capabilities of ArcGIS API for Flex such as clustering with flares, info window rendering on clicks, custom function referencing in symbols, I was able to quickly build an application to display the sentiments. You can see the application in action here. Hover over a cluster to flare it and click on a flare element to see its details. To make things more understandable, I reversed the factor value displayed in the info window. Warning: These are real tweets with sometimes very offensive language, so…. do not call HR on me, ok ? And like usual, you can see the source code here.
BTW, Check out GeoTagger for ArcGIS Runtime to see MetaCarta in action.

Tuesday, March 20, 2012

DnD File using HTML5 into Flex Web App

So….I always wanted to Drag and Drop an external file into a Flex application running in the browser. Well.. you can use the FileReference API to browse and load a file, but I really wanted to DnD. Unfortunately, this is not allowed due to security constraints of the Flash Player. However, using the latest proposed W3C DnD API and the File API, I should be able to use the last two and the AS3 ExternalInterface API to accomplish what I want. So, here is the content of a simple xml file that I would like to DnD and render on a map:
<markers>
    <marker x="45" y="45" label="M 1" info="This is M1"/>
    <marker x="-45" y="-45" label="M 2" info="This is M2"/>
</markers>
I modified the HTML wrapper to use the JavaScript DnD and File API (if available) to listening for “dragenter”, “dragover” and “drop” events on the FlashPlayer container.
var DnD = {
loadHandler:function () {
  var dropContainer = document.getElementById("DnDApp");
  dropContainer.addEventListener("dragenter", function (event) {
    event.stopPropagation();
    event.preventDefault();
  }, false);
  dropContainer.addEventListener("dragover", function (event) {
    event.stopPropagation();
    event.preventDefault();
    event.dataTransfer.dropEffect = 'copy';
  }, false);
  dropContainer.addEventListener("drop", function (event) {
    event.stopPropagation();
    event.preventDefault();
    var files = event.dataTransfer.files, len = files.length;
    for (var i = 0; i < len; i++) {
      var file = files[i];
      var fileReader = new FileReader();
      fileReader.onload = function (event) {
        dndApp.drop(event.target.result);
      }
      fileReader.readAsText(file);
    }
  }, false);
}
};
if( window.File && window.FileReader){
  window.addEventListener("load", DnD.loadHandler, false);
} else {
  alert('Your browser does not support File/FileReader !');
}
On dragenter, I stop the event propagation and prevent the default behavior. On dragover, I do the same and in addition, I update the drop effect to show a “+” icon over the drop area. And finally, on drop, I iterate over the list of dropped files, whereupon I read as text each file using the FileReader API. When the file is read (remember, this is all asynchronous), I hand over the content to the Flex application. On creation completion of the Flex application, the “drop” callback is registered using the EternalInterface enabling the host javascript wrapper to invoke the internal dropHandler function.
private function this_creationCompleteHandler(event:FlexEvent):void
{
  ExternalInterface.call("setObjectID",
    ExternalInterface.objectID);
  ExternalInterface.addCallback("drop", dropHandler);
}

private function dropHandler(text:String):void
{
  const doc:XML = new XML(text);
  for each (var markerXML:XML in doc.marker)
  {
    const mapPoint:MapPoint = new WebMercatorMapPoint(
      markerXML.@x,
      markerXML.@y);
    arrcol.addItem(new Graphic(
      mapPoint,
      null, {
        label: markerXML.@label,
        info: markerXML.@info }));
  }
}
The latter accepts a String argument that is converted into an XML instance, and using E4X, each child marker element is converted to a Graphic that is added to a graphic layer graphic provider array collection. Cool, eh ? Note that the graphic layer has it infoWindowRenderer property defined, is such a way that if you click on any of its graphics, a info window content will be displayed whose content is an instance of the defined component. Like usual all the source code is available here. Have fun DnD’ing.
You can see the application in action by download the markers.xml file, and drag and drop it on the application running here.
PS: As of this writing, the two JS APIs work in Google Chrome 16 and later, Firefox 3.6 and later, Safari 6 will support the standard File API and our favorite (Not!) Internet Explorer 10 (Preview 2+). One of these days, will come back to this and use something like Dojo DnD to abstract me from all this - that will be a nice post!

Monday, March 19, 2012

Offline TPK viewer in a Flex Mobile Application

In this post I will demonstrate how to write a Flex mobile application that displays a map with a tiled layer, where the tile source is the unpacked content of a tpk file. A tile package or tpk is generated using ArcMap and is a zip file that contains a couple of configuration files holding tile metadata and a set of ISAM files where the variable length files contain the tile images. The mobile application enables a user to download one or more tpk files from a web server to the local storage and disconnect from the “network” for offline viewing. When the user selects a local tpk, it will be unpacked and a map will render a tiled layer whose tile source is the local file unpacked images, enabling the user to zoom in and out and pan over the available levels as defined in the metadata file. This is possible due to the availability of the ByteArray class in AS3 and the capability to read and write binary files. The application is based on the Holistic Framework and is linked with the fast airzip library and the ArcGIS API for Flex. The TPKLayer in the map view is a subclass of the Esri TiledMapServiceLayer class where the getTileURL function is overridden to return a URLRequest instance with a custom ‘data’ scheme. This ‘data’ scheme informs the super class that the tile data is in a byte array referenced by the URLRequest instance ‘data’ property. In the TPKLayer, the getTileURL is invoked with a level, a row and a column value that are used to seek using the File API to a specific location to read the binary image at that location into a byte array. Using an iOS deployment, a user can drag and drop a tpk file onto the application's iTunes document folder for a later sync. This is possible due to the UIFileSharingEnabled declaration in the xml application manifest (CaheApp-app.xml). Like usual, all the source code is available, and you can download it from here. BTW, I was able to build the app using FlashBuilder 4.6 and the ArcGIS API 2.5 and upcoming 3.0 for Flex without issues. Have fun and keep me posted on your implementations.