Get some data and import it

Grab populated places points

wget http://edcftp.cr.usgs.gov/pub/data/nationalatlas/citiesx020.tar.gz

Grab US counties map:

wget http://edcftp.cr.usgs.gov/pub/data/nationalatlas/countyp020.tar.gz

Lots more data available in the National Atlas

Extract shapefiles from the two archives

tar -xzvf citiesx020.tar.gz
tar -xzvf  countyp020.tar.gz

Ogrinfo the files to have a look at them

ogrinfo citiesx020.shp
INFO: Open of `citiesx020.shp'
      using driver `ESRI Shapefile' successful.
1: citiesx020 (Point) 

ogrinfo countyp020.shp
INFO: Open of `countyp020.shp'
      using driver `ESRI Shapefile' successful.
1: countyp020 (Polygon)

Notice that the cities shapefile is a point layer and the county layer is a polygon layer.

Import the two shape files into your PostGIS db

To get started with PostGIS see my first tutorial.

ogr2ogr -f "PostgreSQL" -s_srs EPSG:4269 "PG:dbname=mydbname" countyp020.shp -nln us_counties
ogr2ogr -f "PostgreSQL" -s_srs EPSG:4269 "PG:dbname=mydbname" citiesx020.shp -nln us_cities

Make sure you change mydbname to your database’s name.

Checking the Import

psql mydbname
mydbname# \d us_cities
                                   Table "public.us_cities"
    Column    |       Type       |                          Modifiers
--------------+------------------+-------------------------------------------------------------
 ogc_fid      | integer          | not null default nextval('us_cities_ogc_fid_seq'::regclass)
 wkb_geometry | geometry         |
 citiesx020   | double precision |
 feature      | character(27)    |
 name         | character(48)    |
 pop_range    | character(21)    |
 pop_2000     | numeric(8,0)     |
 fips55       | character(5)     |
 county       | character(55)    |
 fips         | character(5)     |
 state        | character(2)     |
 state_fips   | character(2)     |
 display      | numeric(1,0)     |
Indexes:
    "us_cities_pk" PRIMARY KEY, btree (ogc_fid)
    "us_cities_geom_idx" gist (wkb_geometry)
Check constraints:
    "enforce_dims_wkb_geometry" CHECK (ndims(wkb_geometry) = 2)
    "enforce_geotype_wkb_geometry" CHECK (geometrytype(wkb_geometry) = 'POINT'::text OR wkb_geometry IS NULL)
    "enforce_srid_wkb_geometry" CHECK (srid(wkb_geometry) = -1)

You can see that ogr2ogr has imported the shapefile, created a primary key (ogc_fid) and created a spatial index. What you can’t see is that ogr2ogr was also nice enough to add an entry to the geometry_columns table so that postgis tools know that there is a geometry column (wkb_geometry) in this table.

Looking at the Data

Lets fire off qgis and see what we have

Next time, querying with spatial predicates….I promise it won’t take six months for the next post.

sudo apt-get install gdal-bin

Once you have GDAL/OGR installed you get a slew of command line utilities, I’ll try to cover some others in later tutorials, but for now we’re interested in ogr2ogr. ogr2ogr converts between the vector formats that OGR understands.

Example using a shapefile at the city of chicago website:
Download the data

wget http://egov.cityofchicago.org/webportal/COCWebPortal/COC_ATTACH/TIFS2008.zip

Unzip it

unzip TIFS2008.zip

Now we have to do the hard part (not really that hard, but important), look for the projection information in the metadata. I looked in the tifs.shp.xml file and found that the projections is:

NAD_1983_StatePlane_Illinois_East_FIPS_1201_Feet

Google uses WGS84 spatial reference system.
Now we have to lookup EPSG codes that OGR understands for these projections. A good spot to do this is spatialreference.org. EPSG codes provide a short form of expressing projection and spatial reference information.
Once we have that all sorted out we’re ready to run ogr2ogr:

ogr2ogr -f "KML" -s_srs "EPSG:102671" -t_srs "EPSG:4326" tifs.kml tifs.shp

The -f “KML” specifies that we want the output in KML. -s_srs is the source (tifs.shp) spatial reference system and -t_srs is the target spatial reference system found at spatialreference.org, then we specify the output file tifs.kml and the input file tifs.shp. That is it!

Lets shrink it down into a kmz (compressed kml) so that it takes up less disk space.

zip tifs.kmz tifs.kml

To get started in ubuntu 7.04 (should work in 7.10 and debian too):

$ sudo apt-get install postgis postgresql-8.1-postgis

Next we’ll start setting up the PostGIS environment.

Then we need to set up a new database, I called mine geodb:

$ createdb geodb

We then need to bind it to the PostGIS libs with the following:

$ createlang plpgsql geodb

And finally add the PostGIS Tables:

$ psql -d mytestdb -f /usr/share/postgresql-8.1-postgis/lwpostgis.sql
$ psql -d mytestdb -f /usr/share/postgresql-8.1-postgis/spatial_ref_sys.sql

Phew….that was a bit of work.

Now we’re ready to go. We have a database called geodb and we’ve told PostgreSQL that it is a PostGIS database.

So now we can use the shp2pgsql command to import a shapefile into the db:

$ shp2pgsql -W LATIN1 -D -I myshapefile.shp mytablename | psql geodb

-W is for setting the encoding type, I had problems with many shapefiles as my locale is UTF8 and PostgreSQL seems to default to the locale. For me LATIN1 has been working better.

-D sets the SQL output to be in dump format for insertion into the database.

-I flags it to compute the spatial indexing upon import. For large complex vector features this can take awhile, so use with discretion.

All of this is piped to

This gives us a new table called “tablename”, which we can see by doing the following:

$ psql geodb
Welcome to psql 8.1.8, the PostgreSQL interactive terminal.
 
Type:  \\copyright for distribution terms
       \\h for help with SQL commands
       \\? for help with psql commands
       \\g or terminate with semicolon to execute query
       \\q to quit
 
geodb=#

At the PostgreSQL prompt you can enter various SQL commands including queries.

geodb=# \\d mytablename

shows the columns and data types of the data in the table.

Notice the column called: the_geom

This is where the GIS Objects are stored.

The sample query there would show all the data in the shapefile’s associated .dbf as well as a column called

Next time we’ll launch into some querying fun.

Onward to Part II…

(Click for Larger Version 1.9MB)

I have been hacking on a bunch of convenience code in python for the LTM (Land Transformation Model) which serves as the land use projection portion of my research. Basically just wrapping a bunch of ArcGIS functions that do what I need done and then some file manipulation and config file writing before I launch code written by our postdoc. Going to try and do some time projections for all of the state of Indiana and then go back to my Muskegon study area with the new tools to re-run the model up there. It should make things run much more smoothly.

It is not without caveats, mostly security caveats. I work in a firewalled environment and would not recommend exposing the entire arcgisscripting geoprocessing object in a non-firewalled environment, without adding some security. With the client code I am able to write scripts that work on either Win32 or Linux with the same code.

Additionally, you will need to refer to files and directories in your scripts as they would appear to the server instance (with its windows permissions). As ArcGIS scripting routines do not directly allow the passing of GIS data to them, you will have to point to them on the server’s filesystem, geodatabase or remote ArcSDE server as you would locally on the server.

Server:

from SimpleXMLRPCServer import SimpleXMLRPCServer
import arcgisscripting as agis
geop = agis.create()
server = SimpleXMLRPCServer(("10.0.0.128",8000))
server.register_instance(geop)
print "Starting server..."
server.serve_forever()

Client:

#!/usr/bin/python
REMOTE_SERVER = "http://10.0.0.128:8000"
import sys
print "We're running on: ", sys.platform
if sys.platform == "win32":  # We're on win32, call instance of gp object as we normally would.
        print "Importing the geoprocessing object"
        import arcgisscripting as agis
        gp = agis.create()
else:  # Run our calls over XMLRPC to REMOTE_SERVER
        from xmlrpclib import ServerProxy
        print "Using XMLRPC to connect to:", REMOTE_SERVER
        gp = ServerProxy(REMOTE_SERVER)
# Just check that we can call the geoprocessing object.
print gp.ProductInfo() #Show which product we're running just to make sure it is working.

Use:
On the server run:

python server.py

Starting server...

On the client call the Client code at the top of your geoprocessing script, then refer to the gp object as you would normally.

Geopy lets me code a geocoder in 6 lines of python using the google maps API.

Example:

#!/usr/bin/python
from geopy import geocoders
import sys
g = geocoders.Google('ENTER YOUR API KEY HERE')
address = sys.argv[1]
place, (lat,lng) = g.geocode(address)
print "%s: %.5f, %.5f" % (place,lat,lng)

Then run from the command line like so:

# geoloc "500 Main St. 42367"
500 Main St, Powderly, KY 42367, USA: 37.23329, -87.16244

Nice!

I often like to be able to generate a quick histogram of an ASCII raster file that I use for some of my models. I can do this inside of ArcGIS on a GRID file, but I like to be able to do this directly on an ASCII sometimes. Python to the rescue…

In a few lines of python I whipped this up:

#!/usr/bin/python
# By Jonah Duckles jonah@purdue.edu 7/17/2006
# Histogram Ascii will take a standard ESRI ascii file and output a histogram
#   of the values found in the file.

import sys, string, os
from operator import *

file = sys.argv[1]
histo = dict()
fh = open(file)
currow = 0
for line in fh.xreadlines():
    if currow > 6:
        line = string.rstrip(line, '/n')
        row = line.split(" ")
        row = row[:len(row)-1]
        for cell in row:
            if cell in histo:
                histo[cell] = histo[cell] + 1
            else:
                histo[cell] = 1
    currow += 1

k = histo.keys()
k.sort()

print "#Histogram for " + file
for element in k:
    print str(element) + "\t" + str(histo[element])

Now what if I want to be able to do this directly on a GRID file on Linux with no ArcGIS. I admit this is a complete hack, but here it is. A script that calls GDAL bin tools to convert the GRID to ASCII (it does it pretty fast actually), then calls hasc.py above and saves the histogram to a file.

#!/bin/bash
tmp=00-$1.asc
out=hist-$1.csv
gdal_translate -of AAIGrid $1 $tmp
echo "Computing histogram"
hasc $tmp > $out
echo "Histogram output to $out"
echo "Cleaning up"
rm $tmp

EDIT: Fixed a minor bug per pyther’s comment.