ESRI to the rescue: Back in January ESRI launched their professional certification program. I am reporting on my experience taking the exam, in case anyone considers pursuing this certification. (Another candidate experience is over at the ESRI Australia blog.)

To become a ESRI Certified ArcGIS Desktop Professional all you need to do is to pass a test. First good point:  you can take the exam in one of thousands of Pearson Vue test centers worldwide. The test fee is a reasonable 225US$, and ESRI surely hope to cash in on their new preparation course.

The exam  it is a multiple choice exam – 90 ArcGIS questions in 2 hours. You don’t have access to the ArcGIS software or any other material during the exam.

I took the exam in the Manchester, UK, and was directed to a dodgy test center in the commercial wastelands aka ‘vibrant business district’ – not a good reference for ESRI’s testing contractor Person Vue.

The exam itself held some surprises:

1. It is not easy.  The easier questions are like: “A polyline Shapefile has a LENGHT field, and you import the Shapefile into a geodatabase. How is the LENGTH field treated?” Not ArcGIS 101. (This is not an actual exam question.)
2. It requires detailed knowledge of the ArcGIS Desktop applications, geodatabases and extensions, specifically Network and Spatial, but also many others. I suspect many candidates will struggle with this breadth and depth of knowledge – most of us use only specific procedures and one or two extensions, depending on our respective industries.
3. Quite a bit about Python – at least more than I expected.

I can see some room for quick improvement:

1. Questions regarding geomatics, coordinate systems and projections have a strong bias towards the United States. How am I supposed to know which would be the best central meridian to project Louisiana? I can tell for most of Europe, but not for the US.
2. Some answer options are slightly ambiguous and unclear.

My main criticism is that the exam requires the candidate to know almost all there is to know about ArcGIS Desktop by heart. In real life we rather know our most frequently used tools, and everything else and the details we look up as we go along; so I find the exam a bit academic.

On the other hand it does a good job in not only thoroughly testing your knowledge of the ArcGIS products, but also of underlying GIS principles – e.g. if you don’t understand coordinate systems or geoprocessing in detail you are lost.

So should you pursue the ESRI certification? It didn’t take the GIS world by storm – many users seem to wait and see if it gains popularity before splashing out for preparation courses and exam fees. The good point is that the exam is far from being trivial, so passing it actually means that you have a REALLY good knowledge of ArcGIS. In my opinion, if you are after a technical certification, it is your best option and worthwhile.  If you are after a non-technical GIS certification you are more likely to end up with the GISP.

In ArcGIS the Geostatistical Analyst extension provides interpolation tools. You can call start them either from the Toolbox or from the Geostatistical Wizard:

This provides various interpolation methods, but which one should you use?

First, there is not a single best interpolation technique – it depends on what you want to find out and on the characteristics of your data. For example, if you have many outliers or very irregularly spaced sample points, some interpolation techniques are more suitable than others.

An overview of techniques available in ArcGIS Geostatistical Analyst:

• Deterministic techniques. These can be distinguished:
  • By spatial scope:
    • Local deterministic techniques (IDW, Local Polynomial, and Radial Basis Functions)
    • Global deterministic techniques (Global Polynomial)
  • By sample point fit:
    • Exact (IDW, Radial Basis Functions)
    • Inexact (Local Polynomial, Global Polynomial)
• Geostatistical techniques (Kriging), these are only necessary if you want to assess the quality (error) of your interpolation

Start with considering if you need to assess the quality of your interpolation. If not, look only at the deterministic methods.

Within the deterministic methods, decide if you need sample point fit (the interpolation result going exactly through your existing values).

• If you have accurate and precise data, e.g. CO2 level at known point locations, you would typically answer with yes. This narrows the interpolation options down to IDW and Radial Basis Functions.
• If your existing values are not exact sample points, but generalizations by administrative units, an inexact fit is more appropriate than an exact one. This leaves you with the Local Polynomial and Global Polynomial interpolators.

Next, look at the spatial scope of your interpolation: should the interpolation take into consideration only existing values in a specific neighbourhood of the interpolated value?

• If you are dealing with something like CO2 emissions you could argue that every emission source affects the values all over the study area, so a global technique is adequate. This leaves you with the Global Polynomial interpolation as the technique of choice.
• If your data has only spatially limited impact, e.g. household spending on council tax, which do not affect anything beyond the council area, then you would use one of the Local deterministic techniques.

…and so on. This is not a comprehensive decision tree, but you get the idea how to decide which interpolation to use.

In general, the decision for a specific method depends not only on the characteristics of your data, but even more on your research design.

A good starting point to learn more about interpolation techniques in ArcGIS 9.2 is the ArcGIS documentation at:
http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=Overview_of_the_interpolation_techniques_implemented_in_Geostatistical_Analyst

More documentation about the Geostatistical Analyst tools:
http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=welcome, at the left go to Extensions > Geostatistical Analyst

like this:

Imagine the background map being streamed directly into ArcGIS, with no data to download and process, and generalization  automatically adapting to your zoom level, just like in Google Maps!

Such a map service is now available, and it is free for UK academic use. The dataset is called OS OpenData, and it is provided to UK academia through Digimap OpenStream, part of the excellent EDINA Digimap services. In technical terms it is an Web Map Service (WMS), compliant with the OCG WMS specifications.

To use this service you need to register for it and get your personal API key on http://openstream.edina.ac.uk

Once you have your API key, launch ArcCatalog and define OpenStream as a data source:

1. In the catalog tree go to GIS Servers and double click Add WMS Server.
2. For URL enter:
   http://openstream.edina.ac.uk/openstream/wms?token=[your_aipkey]
3. Click Get Layers to verify that it works. Click OK.
4. You now have a new data source entry under GIS Servers, called ‘OS OpenData on openstream.edina.ac.uk’

Now add the OpenData data to your ArcMap project:

1. In ArcMap go to File > Add Data.
2. Navigate to GIS Servers > OS OpenData on openstream.edina.ac.uk > OS Open Data and click Add.
3. If you don’t see any map content, expand the OS Open Data layer, right click  any layer and Zoom To Layer. Then zoom to your are of interest.
4. It always is a good idea to set the coordinate systems to British National Grid.

That’s it!

For further documentation and help see http://openstream.edina.ac.uk

Kudos to EDINA, this really makes work a lot easier for many projects.

Two notes:

• In the same way you can add any other map service that complies with OGC WMS standards.
• Digimap OpenStream is an OGC 1.1.1 compliant WMS, so you can  display the OS map not only in ArcMap, but in any compliant WMS client, like ERDAS Imagine, Google Earth, QGIS, your web application,…  For a complete list of compliant clients see the OGC website.

In Detail:

The ArcGIS licenses that we use on campus are managed by three central license servers. The license server software is the ArcGIS 9.3 (Flexnet-based) license server, the hosts are 27013@alecto.mcc.ac.uk;27013@megaera.mcc.ac.uk;27013@tisiphone.mcc.ac.uk

Now we encounter a strange problem:

On the clients we install ArcGIS Desktop 9.3.1 and set the license server to the above string, using the Desktop Administrator. Desktop Administrator does not accept the setting and returns: “The computer you chose is not a valid license server or is running an older version of the License Manager.”

ESRI says this error is produced when the network lookup by way of hostname or IP address fails. (http://resources.arcgis.com/content/kbase?fa=articleShow&d=33925).  To verify this I run a port scan using Angry IP Scanner, but it shows that the port@host is perfectly available.

I could call ESRI support but can’t be bothered, since fortunately there are two other ways to specify a license server, and both work for us:

• put the license server string into the registry at HKLM\SOFTWARE\ESRI\License, as described in above link
• Set an environment variable ARCGIS_LICENSE_FILE = 27013@alecto.mcc.ac.uk;27013@megaera.mcc.ac.uk;27013@tisiphone.mcc.ac.uk

These are actually very useful, since we can deploy them automatically – a nice side effect!

This resolves the problem.

Backup strategy

In an ideal world we would now define and specify the risk of data loss, then come up with a strategy to mitigate these risks, and finally build a comprehensive solution for backup and disaster recovery, possibly including redundant hardware, off site backup, and other fancy things. This requires knowledge, time and money.

In reality we don’t have all that. Instead, we have an external USB drive. So we will set up the system to perform a nightly backup of important data to this external USB disk.

Prepare the backup drive

We connect the drive to the server via USB and power it up. By default, in Ubuntu Server disk drives do not auto-mount, so we don’t see it in the file system for now. We check the partition table:

sudo fdisk -l

This tells us it connects as as /dev/sdb1.

The drive comes NTFS formatted, and it actually is not a bad idea to leave it as NTFS – this way in case of an emergency we can plug it into a Windows PC and restore files there. To use NTFS on Ubunto we need to install an NTFS driver:

sudo apt-get install ntfsprogs

See www.linux-ntfs.org for details, we use this driver and toolset rather than the Ubuntu recommended ntfs-3g for no apparent reason.

We now can format the drive as NTFS:

sudo mkfs.ntfs /dev/sdb1

We create a mount point

sudo mkdir /media/backups

And allow access for root only

sudo chmod -R 700 /media/backups

A quick test to see if we can mount the drive and write to it:

sudo ntfsmount /dev/sdb1 /media/backups
sudo touch /media/backups/test.file
sudo ls -al /media/backups/*
sudo rm /media/backups/test.file
sudo umount /media/backups

Looks good.

The drive will generally stay unmounted and be mounted only for the duration of the backup operations, so we don’t need to create an fstab entry.

Install backup software

Our main concern are the PostgreSQL and MySQL databases running on the server: the database files cannot be backed up in operation, so we have to either stop the database servers during backup (not good!) or dump the databases and back up the dumps (better!).
After looking around a bit, it looks like Backupninja is a suitable tool to achieve this.

Install Backupninja:

sudo apt-get install gawk rdiff-backup gzip hwinfo debconf-utils backupninja

Installation also creates a cron job for backupninja.

Configure backup options

Create config file from template:

sudo cp /usr/share/doc/backupninja/examples/backupninja.conf /etc/backupninja.conf

All settings in the config file are reasonable.

The actual backup procedure is a series of actions, defined by files in /etc/backup.d. Our first action in the backup process is to mount the backup disk. Therefore we create /etc/backup.d/10.sh, containing

ntfsmount /dev/sdb1 /media/backups

The second action is to backup the system. For this we use the backupninja template:

 sudo cp /usr/share/doc/backupninja/examples/example.sys /etc/backup.d/20.sys

The third action is to backup the package list. For this we use the backupninja template:

sudo cp /usr/share/doc/backupninja/examples/example.sh /etc/backup.d/30.sh

The fourth action is to backup the MySQL databases. For this we use the backupninja template:

sudo cp /usr/share/doc/backupninja/examples/example.mysql /etc/backup.d/40.mysql

The fifth action is to backup the PostgreSQL databases. For this we use the backupninja template:

sudo cp /usr/share/doc/backupninja/examples/example.pgsql /etc/backup.d/50.pgsql

The sixth action is to backup the file system. For this we use the backupninja template:

sudo cp /usr/share/doc/backupninja/examples/example.rdiff /etc/backup.d/60.rdiff

The config and action files must not be readable by non-root users:

sudo sh chmod go-rwx /etc/backupninja.conf /etc/backup.d/*

The settings from the templates are very reasonable, only the rdiff action (/etc/backup.d/60.rdiff) requires a few edits to create a local instead of the default remote backup:

[dest]: type = local
directory = /media/backups
# host = backuphost
# user = backupuser

Finally we need an action that unmounts the USB drive at the end of the backup procedure. We create /etc/backup.d/90.sh, containing

umount /media/backups

Now we test it:

sudo backupninja -t -n

If everything is okay we can now wait for the initial backup to happen tonight at 1am, or initialize it manually:

sudo backupninja -n

Let’s have a look at the log:

cat /var/log/backupninja.log

Lessons learned

• We now have reasonable protection against data loss. However, other risks remain:
  • Destruction of both the server and backup disk, e.g. by fire
  • Theft of both server and backup disk
  • Server hardware/software failure and the resulting downtime during restoration
• Technically our backup solution is way from perfect, e.g. it does not check if /etc/sdb1 is plugged in.
• At least I practiced my Linux skills
• Backupninja is a simple, yet powerful backup solution. In a case like mine a shell script might be more straightforward (mount disk, dump databases to disk, tar filesystem to disk, unmount disk), and easier to customize.
• I might change the backup process to run during daytime. This will allow the user to simply plug/unplug the backup disk and keep it offsite over night.

Productivity Suite is a bundle of tools produced by ESRI UK.  It is installed as an extension on top of ArcGIS. We use it to load UK Ordnance Survey data delivered in a GML format into ESRI Geodatabases.

In principle the installation sequence is:

1. Get a PC with Windows XP and the latest  Windows XP Service Pack (in my case SP3)
2. Install ArcGIS Desktop 9.2. Select the ‘Complete’ option to make sure all components are installed
3. Install the latest  Service Pack for ArcGIS 9.2 (in my case SP6)
4. Install Productivity Suite 1.3.1

Steps 1-3 did go well.  But when I launched the Productivity Suite installer it came up with an error message saying the .NET Support features for ArcGIS are not installed, these are required for Productivity Suite, therefore it cannot install Productivity Suite. I am stuck.

Now it becomes technical – if you are only interested in resolving the installer problem you can ignore the text in italics.

What was going on? I had done a ‘Complete’ ArcGIS installation, which I assumed would also install the .NET Support. It turns out ArcGIS only installs the .NET Support features if the .NET framework is present on the target PC.  I had a freshly installed Win XP with no .NET framework, so ArcGIS did not install its .NET Support features – even with a ‘Complete’
installation.

So I need to install a .NET framework, then install the .NET Support for ArcGIS, and finally should be able to install Productivity Suite.

First we install a .NET framework. ESRI supplies the .NET 2.0 framework with ArcGIS, this is good enough.

From the ArcGIS 9.2 CD run the file \dotnet\dotnetfx.exe.

Now we install the .NET Support features for ArcGIS. These are part of the ArcGIS setup. However, when I go to Add or Remove Programs > ArcGIS Desktop > Change, the setup runs, but does not offer to add the .NET support.

Okay, I decide to start from scratch and uninstall ArcGIS. Then I start a new installation – and surprise, it still does not offer installing .NET Support!  Is there some registry entry that makes the ArcGIS installer believe .NET was still not installed? Anyway, i resort to RTFM and install it from the command line:

Go to Start > Run, type cmd and hit OK. A Command Window opens.

In the Command Window type:
(all on one line, replace D: with your CD drive letter, and keep upper/lowercase)

msiexec /i D:\desktop\setup.msi /qb ADDLOCAL=Applications,.NET_Support,Extensions SOFTWARE_CLASS=Professional

This reinstalls ArcGIS, forcing the .NET Support features to be installed. (And setting it to ArcInfo configuration.) For details on the parameters see the installer documentation on the ArcGIS CD.

Now launch the Productivity Suite 1.3.1 installer… and viola! It installs without any problems!

I am setting this up as a development environment for web applications on behalf of Richard Kingston.

PostgreSQL/PostGIS has a massive learning curve, so I expect to learn a lot in the process. (For now I am quite happy that it is only for development, not as a production environment.) The following is a quick documentation of what I did to set up the PostgreSQL/PostGIS server.

Step 1: Install PostgreSQL/PostGIS

I installed PostgreSQL 8.3 and PostGIS 1.5.3 on Ubuntu Server 9.10, and it couldn’t be easier – just type at the command prompt:

sudo apt-get install postgresql-8.3-postgis postgis php5-pgsql

Done!

We will need a text editor, and I can’t be bothered with vi, so I install Midnight Commander (mc):

sudo apt-get install mc

Oh, and I install pgAdmin on my Desktop PC, plus the Shapefile loader for PostGIS. This will make working with databases and loading spatial data a lot easier later on.

Step 2: Change password for user postgres

The PostgreSQL installation automatically created two user acconts: a Linux user named ‘postgres’, and a PostgreSQL database user (in PostgreSQL speak: a role) also named ‘postgres’. Both are created with blank passwords. Of course we change the passwords.

First we change the password for the database user postgres on the database template1:

sudo -u postgres psql template1

ALTER USER postgres with encrypted password '<newpassword>';

\q

Then we change the password for the Linux user postgres:

sudo passwd -d postgres

sudo su postgres -c passwd

Step 3: Configure PostgreSQL

We need to change some configuration settings for the PostgreSQL server. All the following settings are in /etc/postgresql/8.3/main/postresql.conf, so we open it for editing:

cd /etc/postgresql/8.3/main
sudo cp postresql.conf postresql.conf.original
sudo mc -e postresql.conf

By default PostgreSQL only listens for local connection requests (127.0.0.1). If we want to connect from clients on other machines, e.g. pgAdmin on my desktop PC, we need to listen for connections on other network adapters:

listen_addresses = '*'

For security we set it to encrypt passwords:

password_encryption = on

Exit Midnight Commander, then restart the PostgreSQL server:

sudo /etc/init.d/postgresql-8.3 restart

Step 4: Set access rights

By default PostgreSQL only allows access from the local machine.  I want to allow access to the databases from remote clients like pgAdmin on user’s PCs. However, this should be only through SSL and from PCs within the university network.

We need to edit access settings in /etc/postgresql/8.3/main/pg_hba.conf, so open it for editing:

cd /etc/postgresql/8.3/main
sudo cp pg_hba.conf pg_hba.conf.original
sudo mc -e pg_hba.conf

At the end of the file we add a line specifying that SSL access is allowed from within the university subnet.

# TYPE DATABASE USER IP-ADDRESS IP-MASK METHOD
hostssl   all      all  150.67.0.0/16  md5

Now I should be able to access the PostgreSQL server from pgAdmin on my Desktop PC … just a second … Yep, works!

Step 5: Make a database template

Every time you create a database in PostgreSQL you will have to specify a number of parameters (data types, spatial reference systems, etc). To facilitate database creation, we can define a database template, and later on we just base new databases on the template. To create a template for geospatial data we do:

Switch user to postgres:

sudo su postgres

Create a new template database and load the PostGIS settings into it:

createdb postgistemplate
createlang plpgsql postgistemplate
psql -d postgistemplate -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
psql -d postgistemplate -f /usr/share/postgresql-8.3-postgis/spatial_ref_sys.sql

Test the template:

psql -d postgistemplate -c ""SELECT postgis_full_version();""

Exit user postgres:

exit

Now we have a template ‘postgistemplate’ which contains everything a geospatial database needs. We will base future geospatial databases on this template.

Step 6: Set up a multi user PostgreSQL environment

Until now we only have the default user account ‘postgres’ to access the database server, but I want all of our developers to have an individual account, so they can all use the server, but not mess with each others databases. Here the PostGreSQL concept of roles (users and groups) comes in handy.

First I create a PostgreSQL group role for staff. I can accually do this from my Desktop PC:

1. Using pgAdmin, connect to the server as user postgres
2. Add group role ‘staff’. Privileges: Inherit, Create objects, Create roles

Create PostgreSQL role for each member of staff:

1. Using pgAdmin, connect to the server as user postgres
2. Add a new login role <some_user_name>. Privileges: Inherit, Create objects, Create roles. Role Membership: staff

Problems and solutions

When a user logs on using pgAdmin and tries to create a database based on postgistemplate they get: “ERROR: permission denied to copy postgistemplate”.

I thought this was caused by insufficient access rights to the database postgistemplate, tables geometry_columns and  spatial_ref_sys. So I use pgAdmin to set for both tables: Privileges > Add group staff, ALL. However this does not resolve the problem!

Fortunately Nomad Labs’ instructions on creating a PostGIS template help: the database used as a template by non-admin users must be explicitly designated as a template. This is set in the system database ‘postgres’, catalog ‘PostgreSQL(pg_catalog)’, table ‘pg_database’. Here all databases are listed. I go to the database ‘postgistemplate’ and set ‘datistemplate’ = TRUE – And voila, it works!

The animation is actually just a series of images, combined into a single file in GIF format.

You need ArcGIS (I have 9.3) and Corel Photo Paint (X3).

Start with designing your map in ArcGIS:

Go to File > Export and export the map in a raster format, e.g. as frame1.gif.

Then move the marker  a bit (or whatever you want to animate).

Export this frame as well: File > Export, name it frame2.gif. Continue creating individual frames and exporting them. I made 4 frames (frame1.gif to frame4.gif).

Now go to Corel Photo-Paint and load the first image of your animation (frame1.gif).

Click Movie > Create from Document. This creates a movie with a first frame.

Click Movie > Insert from File and add the second frame (frame2.gif). Repeat this, adding all frames.

Go to Movie > Play Movie and see what it looks like. (Hit Esc to stop.)

Go to File > Export and save the movie in ‘GIF Animation’ format, e.g. as mapanimation1.gif. A window with export options comes up – here you set looping (Frame Repetition) and how fast it should run (Frame Delay).

Done!  Now use this animation in your PowerPoint presentation or on your website.

I made this on behalf of Martin Dodge, Geographer at the University of Manchester, who submitted it to the exhibition ‘ Mapping Manchester: Cartographic Stories of the City’. The Exhibition is on at Manchester’s John Rylands Library on Deansgate until January 17 2010.

I used ArcGIS 9.2, census data and a couple of days time to create the animation. ArcGIS does a decent job for the occasional simple flythrough. If you want to do professional 3D animation, like the guys from Arup did for Manchester, you need much better resources in terms of software, data and time. (Envy!)

I did a quick example (click it):

So how does it compare to Google Maps?

• OS OpenSpace is a lot more detailed – it basically contains every footpath in Britain. Most useful for any outdoor activity. Compare for yourself [1]
  
• Licensing: Ordnance Survey claims ownership of any derived data you create using OpenSpace [2]. This has caused a lot of controversy, however, Google has a similar clause [3].

[1] http://wtp2.appspot.com/wheresthepath.htm
[2]http://openspace.ordnancesurvey.co.uk/openspace/developeragreement.html, clause 5.4
[3] www.google.com/accounts/TOS, clause 11