Tuesday, March 9, 2010

Augmented Reality, iPhone and ArcGIS Server

I am a HUGE fan of Augmented Reality, and one of my favorite applications on the iPhone is Layar. In this post, I will "walk you" through how to setup a simple ArcGIS Layar Point Of Interests (POI) layer and how to expose it through the iPhone application. In addition, I created a small Flex application that will enable you to add POIs to that feature class through a python geo-processor task.

BTW - this is heavily geared to using ESRI desktop and server tools and I'm assuming that you are familiar with these tools.

First, request a developer key from http://dev.layar.com/. This might take a couple of hours to get to you.

So, in the meantime, let's get started. Using ArcCatalog, create a new point feature class called 'Layar' in a geodatabase with the following fields: Title (text), Line2 (Text), Line3 (Text), Line4 (Text), Attribution (Text), Type (Integer), ImageUrl (Text:512).

Import that layer into a map document using ArcMap. Save and publish the MXD as a Map Service. Just to be on the safe side, clear your REST endpoint service directory cache and list the available services. You should now see the new map service and you can now query using REST the newly created 'Layar' layer.

Next, we need to populate this 'Layar' layer. With the Advent of ArcGIS 10, this could easily be achieved using the new FeatureTask endpoint. However, I'm still using a 9.3.X server and I will have to rely on a geoprocessing task to add point features to the Layar feature class. For the sake of simplicity and quick deployment, I decided to implement this GP task using python. So, using ArcCatalog, create a new GP Toolbox and add to the toolbox a new python script. You can download the script from here. Again, I'm assuming that you are familiar with all these tools. This script will have two parameters. The first input parameter should be named 'featureSet' of type 'FeatureSet' and its schema should reference the Layar feature class. The second parameter should be named 'objectID'. It has an output direction and is of type LONG. Basically what the script does; it expects a feature set with one feature. That feature is read using a cursor and placed in memory. Next an insert cursor is created on the Layar feature class and populated with the in memory information and executed. Upon a successful execution, the OBJECTID of the last inserted feature is retrieved and returned back as a parameter. The whole script is surrounded by a try: except: in case of an exception, where a -1 as an OBJECTID value is returned back. Simple but not simplistic :-) Publish this Toolbox, clear the cache and now you should have new GPService task.

To use this GP task interactively, I created a Flex based web application.
This application consists of a side-by-side map and form component enabling the user by zooming and panning to click and define POI coordinates and to populate that POI attributes. By clicking the Save button, the above publish GP task is invoked and upon a successful completion, the map is refreshed to show the newly added POI. The Flex application source code can be downloaded from here.

Now that we have created and authored this Layar information, we need to publish and consume it on the iPhone Layar application. The Layar folks have described in detail the JSON publishing format - http://layar.pbworks.com/GetPointsOfInterest. So what we need now, is a middleware that accepts an HTTP request from an iPhone, convert the input parameters to an ArcGIS REST query, invoke a query task whose URL is the Layar layer URL in the above publish map service, read the ArcGIS JSON response and convert to a Layar JSON response. Easy, eh ? That is why we have computers :-) I accomplished all this using Java Web Application - you can download the source code from here. This web application is basically a servlet that relies on the Apache HTTPClient library for http communication and the Jackson JSON processor library for fast consumption and JSON production. You might wonder why I decided to write this in Java versus say python. well...First, I'm a Java nut. Two, I can whip this up fairly quickly, Three, wanted to experiment with Google Java App Engine so I can be "in the cloud man !" - LOL, last one is just an artifact of the first two - but it is cool ! When the iPhone Layar application requests POI, it does so on a predefined URL (more on this later) with the following HTTP GET parameters, lat: your current latitude location, lon: your current longitude location, radius: the search radius in meters plus other parameters. In this application I'm just decoding lat, lon and radius.

Now that you have an in-the-cloud POI Layar endpoint and assuming that you have gotten your Layar developer key, you can sign in to http://dev.layar.com/ and create your own layer. The form will request a POI URL, enter your in-the-cloud poi servlet endpoint. All of the above was leading this point. I highly recommend that you test your layer using the Layar test page http://dev.layar.com/api20test/layarTestPage/.

To test this on your iPhone, go to "Settings" add add your developer ID and key to the Layar application settings. Launch the Layar application, and you should see your newly published layer.

This is just the tip of the iceberg. There is so much more that you can do, adjust and customize. I just wanted to share this "little" experience with you. Hope you found it useful and tell me what you think.


Harley said...

AR is cool. this is interesting for sure, and I am still trying to get the concept... i have an i touch not an Iphone.. but if I had one, iphone has the camera and the layar app provides an image overlay and then the overlay on the image comes from your own arcGIS server and adding points communicates from the iphone to the arcgis server, via layar?

Harley said...

oh, the flex app provides the means to add points. The Geo Processing service is set up via the script you provided... okay.. that's really cool, and I can set up this part for sure. then.. wait until I get an iphone.

ThunderHead said...

Layar just holds the metadata of the later information such as what url to invoke to get the POIs - the iPhone communicates with the middle tear who communicates with AGS - the flex app is just to add POIs to AGS via py script.

Harley said...

Yes.. but just adding points is a pretty big deal... have had to use arcobjects. I need to do more python scripting, so thanks for that.

what is the flexapiutils.swc?

ThunderHead said...

Eh...that is my private flex library with some utils in there ;-)

Harley said...

running into an error when trying to "save" the point. do you think it matters if the geoprocessing task is accessing a personal geodatabase, a file geodatabase, or sde geodatabase?

ThunderHead said...

Make sure the permission are correct on CP script

Harley said...

okay.... a few mistakes on the poi creation for arcgis:
1) might have been a conflict with sdk's I was on 3.2; dropped in 3.5 so no longer was getting the flex error:
(TypeError: Error #2007: Parameter blendMode must be non-null....)

2) i needed to edit the script to point to my geodatabase.

3) had one or two typos on my side so fixed those.

4) checked permissions on the arcgis server, however, this is not enabled, so my assumption is they are not needed. is that correct?

also, checked permissions on the file system and checked groups and those are good for all users.

so the error is: object has no attribute strerror, on the server for the script.. which is good.

ThunderHead said...

Good progress - the problem that I had was that the script was not accessible to AGS - so had to grant SOC access to the script

Harley said...

okay.. I need to try something else to trouble shoot this.

I will try and figure out a simple python script and see if i can figure out what is going on...

oh, does the Layar feature class need to be in a feature data set? the reference to featureSet for the input is confusing.. but I do see this is required in the documenation.

I was able to run the example you provided and add a point to your server... so i must be close

Harley said...

okay.. I have two servers being used to figure this out.. and realized there were more typos in the python script that i thought was the syntax... but now after much reading of python.. it's more obvious..

for this:
cursor = gp.searchcursor(rFC,"","","OBJECTID","OBJECTID D")

mod to this:
cursor = gp.searchcursor(rFC,"","","OBJECTID")

but maybe that second argument is needed.

i had pointed the path to my gdb but left the r and didn't replace \ with / in my path:
rFC = r"C:\MyData\MyGeodatabase.gdb\Layar"

as I am using access gdb should be kind a like this:
rFC = "C:\MyData\MyGeodatabase.mdb\Layar"

But on my first test server i was able to add these and the previous changes... and Success... although my coordinate was stateplane not wgs84..

but should be able to resolve this in the morning..

cool home work.

ThunderHead said...

To test this - use ArcCatalog to author the FeatureClass - next, use ArcMap to author the GP - execute the GP in arcmap first and make sure that u can add points to the feature class. Publish the Toolbox - use the REST directory service to navigate to that task and try to execute it from a browser - verify that the result is in the feature class - and now, u can execute it from the flex app without issues. sorry for all the pain - but that is AGS/GP for u :-(

Harley said...

that's the right work flow. will do.

no pain, no gain.

the AGS/ GP using modeling or scripts, is a better solution than pure arcobjects...

since it limits the people working on a problem to heavy programming and or for flex coding, to solutions like weborb. GP using modeling is more intuitive, and can help get things out of the black box for the clients.

AGS/GP, as well as documentation, has matured, this is a great intro back into it... so thanks for the lesson.

Layars, sign up ask which layers I am developing... ? testing or publishing is the only layers right, or do they want a description of an application?

ThunderHead said...

Eh....do not recall this question :-( for right now, I'm just using it for testing.

Harley said...

what is the REST service expecting as far as format? there is a text box with featureSet (GPFeatureRecordSetLayer) above it.

Harley said...

aah.. json string:
here is the correct format..tested it on your REST GP tool:

this i pulled from the debug console using the tracetarget in the flex code.

so that is good.

ThunderHead said...

Correct - and did it work ? BTW - did u setup the correct connection string in the py script ? and added the .sde file ? make sure the grant SOC access to all that - AAAAAH the joys of working with AGS :-)

Harley said...

if using .mxd to connect, then the .sde file is not needed, if only the data service is created .sde file is needed and applies to all types of gdb. (note: these are often stored by arc catalog in the users name directory somewhere)
Unless someone is well versed in geoprocessing on a server, I found these links useful:
1) Adding a New Service
2) Using Python for the geoprocessing
3 Flex api - geoprocessing tasks
4) All of the server related geoprocessing docs

Harley said...

However, the sde file might be needed to point to the path of the non sde gdb! Otherwise to make the sample app work, I placed all the files (gdb, HTML, swf, py) into one folder (layarapp) at the root of the web site. The py file pathing needs to use \\ foreach \ in the UNC format. If the file name root is a drive letter, an option is to replace each \ with / or backslash.
The basic poi flex app is now working. Yeah!
I started off with a complex web file structure. Now that I switched to a simple web file structure and the app is working, I can work back to a more secure file structure. Nice, I am a fan of geoprocessing on the web. Thanks for helping me.

Harley said...

Just in case I forget, here are the specific primers needed to configure the gp task folder files:
1) Instead of just tossing all files into one folder GP Tool Share Folder Structure use tooldata, scripts, doc, and scratch folders.
2) The GP task automatically creates, for a short time, any scratch spaces on the web server in the jobs folder; where any data is going to be written. it is a similar structure, but a different url, this image
compares GP tool share folders to AGS file
3) The tooldata folder contains either non-sde gdb or connect.sde files
4) I was having trouble with UNC's related to the script and toolbox. what worked for me is, open the toolboxes within arcmap, then test scripts in ArcMap. Copy the toolbox to the mxd location so it is in the web project url, find it in the username/applicationdata/esri/toolboxes folder. more on GP script tools
5) However, it is supposed to work as long as the SOC user is on the machine and has access. this is the toughest part so read:
Data access considerations for geoprocessing tasks

Okay, that wraps it up.

Ala said...


Thanks for the great AR application! I think I also caught your UC2010 flex viewer session in San Diego :-) very motivating!

* the mapservice works, the py gp script runs, but I keep getting the "attribute strerror", which I think has to do with the in_memory workspace..? Any hints..
* Could you elaborate a bit more on the LayarWeb src and what steps (I use Eclips) I need to take to get a webapp running?

BTW: I use ArcGIS 9.3.1 SP2.

Thanks very much for any info!

ThunderHead said...

Glad u liked the UC preso - as to the GP error, not sure what this error is :-( do u have more details - as to creating a web application, I use IntelliJ and has a web deployment environment - I know there is the same in eclipse - u might have to download the web tools platform (http://www.eclipse.org/webtools/)

Paresh Masani said...

My AR application is very simple. Not doing anything fency. I have few Map Points with Latitude, Longitude, title, subtitle, and image. I want to display them on AR View such that they don't overlap with each other when couple of MapPoints are close together. I am struggeling with framing a frame with different starting and ending points but not got around so far. Can you thinkg something that can be useful for me in this regards?

ThunderHead said...

Very interesting question - U can cluster the data on the server side based on their distance and proximity and present clustered marker rather than individual markers. Now not sure, if u can do this, but when u tap on the clustered marker, it will list the cluster details.
Just an idea.

Paresh Masani said...

Sorry but I forgot to tell you that it's an iPhone application. I have been using ARkit ( https://github.com/nielswh/iPhone-AR-Toolkit) but it has the frames overlapping issues. Not able to get around it since long. Not sure where and what to modify...!!

gisjames said...

Hi Mansour, when I add the agslib swc to this flex app, I receive an error that the component SimpleMarkerSymbol is defined more than once in the namespace. Apparently, this component is defined within the flexapiutils.swc as well. How do I remove the redundancy without losing the math functions found in the utils.swc? Thanks!

gisjames said...

Hi Mansour, when I add the agslib swc to this flex app, I receive an error that the component SimpleMarkerSymbol is defined more than once in the namespace. Apparently, this component is defined within the flexapiutils.swc as well. How do I remove the redundancy without losing the math functions found in the utils.swc? Thanks!

Mansour Raad said...

This is a _very_ old app that has not been updated to the latest api - take the "spirit" of the app !