Here is a quick confirmation that you can use Python to draw very detailed maps; using the previously specified method I was unable to get python to draw all UK OAs due to their great number (c.220,000) and high complexity (c.50,000,000) vertices. Additionally I was unable to use the generalised OA boundaries for the UK from UKBorders as they contain topological errors that the shapefile reader cannot deal with. ArcGIS is obviously a bit clever in how it handles bad topologies. So I extracted all the vertices and fed them into shapely polygons, and visualised them in the same way, but without reading shapefiles directly into python and was able to output this:

This method has had an impact on the speed of computation as it can take roughly 25 minutes to output this map. The map looks pretty good, aside from a slightly odd polygon in the Bristol channel. Nevertheless, coupled with the operations that shapely, and other geo-libraries, can do this si increasing indication of the maturity of GIS in a variety of platforms. Oh, and it’s all free!

Thanks to Sean Gillies for commenting on my last post, he put me onto a couple of Python packages that he’s been involved in creating that allow you to do some really excellent geospatial things. The shapely package is a great implementation of a lot of spatial analyses that you can do on projected (i.e. flattened) datasets, including topological operations and a full set of object types. The descartes package allows better integration of matplotlib with spatial data, particularly in terms of not having to use the “fill” plotting function repeatedly, but creating a more efficient set of “patches” which can then be added to the figure plot. The overal impression I got from descartes is that it wasn’t spectacularly different from the method detailed in my previous post, but it gives you more control and stability over the map plotting process; whereas using raw matplotlib you are inclined to hope that the map outputs correctly (it all seems a bit up to chance), using descartes you have a more robust and easily manipulable output.

In order to test this I rewrote my previous thematic map script to: firstly convert the shapefile geometries into shapely polygons, and secondly to pass those shapely polygons to descartes and draw a map plot using descartes-matplotlib. The only slightly odd piece of functionality that I found was that you can’t pass the shapely polygon object a list of shapely points in order to create the polygon, rather you have to pass a list of x,y tuples – much less satisfying!

Nonetheless, the changes were easy to implement, and with the previous script as given basically include:

from shapely.geometry import Polygon

points = []
for i in range(0,number of points in shapefile):
 tempx = float(x coord of point in shapefile polygon)
 tempy = float(y coord of point in shapefile polygon)

 points.append((tempx,tempy))
polygon = Polygon(points)

The above method creates a simple polygon without holes, shapely can accomodate this is need be though. Having created the shapely polygons, all that remains is to create a patch.

from descartes import PolygonPatch

patch = PolygonPatch(polygon, plus colour and line considerations)

Then you simply add the patch to the matplotlib figure you have already created so:

from matplotlib import pyplot

fig = pyplot.figure(1, figsize = [10,10], dpi = 300)   #create 10x10 figure
ax = fig.addsubplot(111)    #Add the map frame (single plot)

# here you create all the polygons and patches

ax.addpatch(patch)   # simply add the patch to the subplot
# set plot vars
ax.set_xlim(get xmin and xmax values from data)
ax.set_ylim(get ymin and ymax values from data)
ax.set_aspect(1)

pyplot.show()

Using these basics I was able to create a basic OAC map using Welsh OAs as an example:

I though I would explore the possibility of creating thematic maps using Python, this post documents my initial attempt. The output is hence rather basic, but encouraging. The primary reason that I wanted to test the mapping potential of python is to allow for some basic automated map production in order to quickly visually assess the geographical patterns contained within large data sets. This is something that I am at a loss to do in ESRI’s ArcGIS, although that might change in ArcGIS 10. For fans of R I know it can be done there, however R is too tricky for me! My colleague James Cheshire explains the method in R here.

The first hurdle in map making is getting the data in, for this I used the shapefile reader that Zachary Forest Johnson put together for his excellent blog ‘IndieMaps.com‘. This allowed me read in any of my masses of pre-existing Shapefile format datafiles, and indeed use the python scripting functionality in ArcGIS to perform spatial operations and then output a map quickly and without the hassle of dealing with ArcGIS layouts.

Once you have download the shapefile reader, it is easily implemented using:

import shpUtils   #imports the shapefile reader
#Load a shapefile into an object called shpRecords
shpRecords = shpUtils.loadShapefile('\filename.shp')

This is undoubtedly simple, what you then have is a (slightly) complex object which contians all of the shapefile data nested as lists and dictionaries. In order to get my head round this I spent some time investigating it, a standard shapefile that contains areal geographies (i.e. UK Output Areas) will have a similar set up to this:

• The first list (shpRecords[i]) records the number of complete geometries, this corresponds to the number of rows in the attribute table. Thus a single polygon has 1 row in the attribute table and 1 list (list index 0) in Python.
• The second dictionary (shpRecords[i]['key']) records two branches, reporting either the ‘dbf_data’ from the attribute table, or the ‘shp_data’ from the .shp file describing the underlying geometry.
• Choosing the ‘dbf_data’ key (shpRecords[i]['dbf_data']) allows you to see the attributes recorded column-by-column for each row (and hence each geometry) in the attribute table. Thus shpRecords[i]['dbf_data']['name'] will return the attribute value for the field ‘name’ for the ith geometry in the shapefile.
• Choosing the ‘shp_data’ key (shpRecords[i]['shp_data']) allows you to access the various components of the shapefile’s geometry. In the case of a polyline/polygon you get dictionary items ‘ymax’, ‘ymin’, ‘xmax’, ‘xmin’, ‘numpoints’, ‘numparts’ and ‘parts’. Clearly the first 6 items are properties of the ith geometry you are querying, so it allows you to form a bounding box, get the number of vertices in the line/polygon, and draw separate lines/polygons if the shapefile is setup to have spatially discontinuous shapes for each row.
• The thing we are most interested in is the ‘parts’ dictionary key, as this contains all the coordinates for the particular geometry being considered, this is accessed as: shpRecords[i]['shp_data']['parts']. The next list (shpRecords[i]['shp_data']['parts'][j]) thus allows you to distinguish between parts in a multipart file. i.e. the jth part of the ith geometry.
• Having come this far, one final dictionary allows us to see the coordinates themselves, this dictionary simply offers us ‘x’ or ‘y’. Thus finding the x-coordinate of the ith geometry and jth part is accessed by: shpRecords[i]['shp_data']['parts'][j]['x'] – simple!

I have been using matplotlib – a python library for scientific visualisation a lot recent, and have found it a very simple and powerful resource, so I thought I’d see if it could be made to draw a map.

Firstly import the pyplot element which does all the figure drawing:

import matplotlib.pyplot as plt

Now lets use the “fill” component of matplotlib to draw all the geometries in a shapefile – my shapefile is Output Areas in Southwark. Firstly we need to loop through each geometry, and then draw a polygon using all the points contained within each geometry. I omitted a loop for multipart geometries as my shapefile has none, however this would be very easy if the data did have multiple parts- simply add a loop in the middle!

for i in range(0,len(shpRecords)):
 # x and y are empty lists to be populated with the coords of each geometry.
 x = []
 y = []
 for j in range(0,len(shpRecords[i]['shp_data']['parts'][0]['points'])):
  # This is the number of vertices in the ith geometry.
  # The parts list is [0] as it is singlepart.

  # get x and y coordinates.
  tempx = float(shpRecords[i]['shp_data']['parts'][0]['points'][j]['x'])
  tempy = float(shpRecords[i]['shp_data']['parts'][0]['points'][j]['y'])
  x.append(tempx)
  y.append(tempy) # Populate the lists  

 # Creates a polygon in matplotlib for each geometry in the shapefile
 plt.fill(x,y)

plt.axis('equal')
# This sets the x and y axes as equal intervals.
# NB this script will only work for projected data, for geographical
# coordinate systems get ready to do some maths  

plt.show() # Draws the map!

This is the simplest form of the script, it will simply draw the shapefile with each area filled a random colour. This is not that useful, but it is easy to create a thematic maps of categorical data, so let investigate a way of doing that. I’ve got data for the Output Area Classification, which is a clustering of areas by social characteristics, I know that there are 7 supergroups in the classification, named numerically, so before all the processing of the shapefile I can create a dictionary of colour choices for each group. I’m using hexadecimal colours that I got from Cynthia Brewer’s website for a ‘qualitative’ 7 class classification. The dictionary looks like this:

oacSGroups = {'1':'#A6761D','2':'#E6AB02','3':'#66A61E','4':'#E7298A',\
'5':'#7570B3','6':'#D95F02','7': '#1B9E77'}

Thus the key ’1′ returns the associated hex colour, this can be linked to the ‘dbf_data’ key in the shapefile. In the plt.fill() component I simply have to specify the colour choice, thus we alter the line in the above script to read:

plt.fill(x,y,fc = oacSGroups[str(int(shpRecords[i]['dbf_data']['supergroup']))]\
,ec = '0.7',lw=0.1)

‘fc’ is the ‘foreground colour’ we are asking python to make the colour equal to the value in the oacSGroups dictionary where the key is the value contained in the attribute table for the ith row in the ‘supergroup’ field. Thus if the ith row had a ‘supergroup’ value of ’7′ that foreground colour would be set to ‘#1B9E77′. ‘ec’ is ‘edge colour’ and ‘lw’ is linewidth, here I have set the values to display fine, light grey lines.

Finally, as basic a map as this will turn out to be, we wouldn’t be anywhere without a legend. The following a a very basic, wholy manual way to add a legend to the map:

p1 = plt.Rectangle((0, 0), 1, 1, fc="#A6761D")
p2 = plt.Rectangle((0, 0), 1, 1, fc="#E6AB02")
p3 = plt.Rectangle((0, 0), 1, 1, fc="#66A61E")
p4 = plt.Rectangle((0, 0), 1, 1, fc="#E7298A")
p5 = plt.Rectangle((0, 0), 1, 1, fc="#7570B3")
p6 = plt.Rectangle((0, 0), 1, 1, fc="#D95F02")
p7 = plt.Rectangle((0, 0), 1, 1, fc="#1B9E77")

plt.legend([p1,p2,p3,p4,p5,p6,p7], ["Super Group 1","Super Group 2",\
"Super Group 3","Super Group 4","Super Group 5","Super Group 6","Super Group 7"], loc = 4)

This simply creates 7 rectangular plots which don’t appear on the plotted output, but instead are passed to the legend creator, each rectangle has the appropriate colour to match the mapped representation, and a label, shown int he legend as two ordered lists. The ‘loc’ tag allows the setting of where the legend will appear, 4 denotes the bottom right corner. the tag ‘title’ allows you to add a title to the legend as a string.

An example output looks something like this:

This took a couple of seconds to produce, and accounts for 846 individual geometries, which actually have quite a number of vertices.

I’ll update the blog should I find new methods to visualise spatial data in python.

Geodemographic classifications are funny things, they report a view of the world which suggests that areas can be split into groups within which all areas share the same or similar characteristics. This is not an inherently bad thing, for large scale analyses it can be a very useful way of simplifying a diverse array of variables into something that characterises the underlying patterns in the distribution of data. However, for smaller scale analyses I am increasingly finding that non-bespoke geodemographics are limited, I attempted to demonstrate this on a national scale by looking at the entropy scores for each OA in the UK with respect to distance from all supergroup cluster centres (here). Recently, Pete Fischer presented some very clever work in this vein at the recent GISRUK 2010 conference, he used fuzzy classification strategies to account for the likelihood that each OA does not fit exactly into any particular grouping, and that different OAs fit differently into the same group. Aidan Slingsby at City also showed this very nicely visually with his ‘OAC Explorer’. Procedings from the conference can be found here.

With this in mind, I wanted to test the variability of the data in Southwark, my study site, with respect to OAC. OAC paints a very flat picture of the population of Southwark as shown below, and had led to me using LOAC, a London specific variant of OAC created by Jacob Petersen a previous research student at UCL and available as a layer on the London Profiler. Using OAC, Southwark is primarily ‘multicultural’, there is more variability in the LOAC classification however, as is evident.

OAC and LOAC classifications for Southwark.

Inspired by some of James Cheshire’s great work with surnames I employed a method called “Multi Dimensional Scaling” or MDS. Multi Dimensional Scaling is great for exploring similarities and dissimilaries in data, rather than clustering data as in the creation of OAC, it reorders it so that similar datapoints have similar values. One of it’s great advantages is that it allows for the scaling of data that has many dimensions, such as the 41 OAC variables, into fewer dimensions representative of those 41, these can subsequently be visualised. Traditional approaches in geography have used MDS to scale many dimensions into 2, using these 2 to adjust spatial coordinates to ‘blow apart’ maps, reordering places that are similar together and dissimilar further apart. Such representations challenge the validity of Tobler’s 1st law – near things are more similar than distant things. In this case however I don’t want to blow up Southwark, so I follow Cheshire’s lead in using the scaling to specify a colour for each area in which similar colours indicate similar areas in terms of OAC variables and different colours represent different areas. I experimented with both greyscale and RGB colour scales for this representation. Firstly though, a note on how I got there:

1. Download the OAC variables from CASWEB, using the ‘recipe’ specified by Vickers et al (2005).
2. Standardise all the variables – I used Z-score without really checking for normality, although in reality this would be preferable – Vickers suggests some other methods of standardisation.
3. Compute a distance matrix for the MDS. This means calculating the similarity of each pair of OAs, given n OAs this thus leads to an n x n size matrix, a size that can rapidly become unmanageable beyond local scales. I used ‘canberra distance’ (an arbitrary choice) to compute the matrix which is given by: where i relates to the value of the first object in a pair and j the second, and k denotes the variable in question.
4. This matrix is then input into an MDS solver, as a python fan I used the fantastic code written using Numpy by Jeremy Stober, although I added to it to do all the standardisation, distance matrix creation etc as part of a logical process.
5. Specifying the number of output dimensions (I used 1 and 3) allows you to reduce the large distance matrix into a vector (1d) or matrix (3d) of values, these can then be scaled between 0 and 255 to be converted into digital numbers for visual display. Thanks to James Cheshire for the ArcGIS script to assign RGB values in Arc.

The results I got from this preliminary exploration were as follows:

MDS Scaling of OAC variables into Greyscale Representation.

This is a very interesting way of looking at the OAC data, as the comfortable uniformity of the seven classes has been lost, instead we can see trends and similarities, but also a fair amount of discountinuity and noise. In the black and white representation a spectrum is presented in which very dark and very light colours are the most dissimilar slowly converging through the spectrum. The resultant mapping clearly displays areas of similarity, the more affluent southern tip of Southwark, the Southbank region in the north of the borough, and the former docklands in the north-west. Counterpoint to these areas is the middle band of Southwark represented by darker hues, and roughly aligned with known areas of deprivation characterised by high-levels of social housing, higher levels of non-white residents, lower level of educational attainment, poorer health etc. What is clear though is that the picture is not uniform as suggested by OAC, and that there exist notable pockets of difference, possibly interpretable as gentrification, particularly around parks. There is also evidence for some fairly notable discontinuities in demographic structure which isn’t immediately obvious in the OAC classification.

I also mapped an MDS output for 3 dimensions onto an RGB colour scale, as below:

MDS Scaling of OAC variables into Colour Representation.

The colour representation should be a more nuanced reading of the similarities and differences, although it is immediately more challenging to interpret. One of the interesting factors is how the southern area of Southern, most characterised by a blue/purple colour has now been distanced from the southbank and former docklands areas, suggesting they are more distinguishably different than previously. The previously dark area is now a pinkish hue, again suggesting a uniformity in that area, however it is flecked with a variety of colours suggesting that deviations in demographics amongst the areas of high deprivation are not similar to each other, but distinct enclaves each with their own specific character.

This constituted a preliminary study, time permitting I will continue to investigate interesting methods such as this. It is however a computationally intensive process, and a treatment of, for example the UK in this manner is out of the question. Nevertheless, I may update it at different scale in the future.

Acknowledgment: Boundaries Crown Copyright 2010 Ordnance Survey. A UKBorders/JISC Supplied Service. Data from CASWeb.

The following map shows an entropy score by Great British Output Areas based on each OA’s ‘distance’ from each OAC supergroup cluster centre. Essentially I’m attempting to measure whether any given OA fits discretely into it’s cluster assignment or not. I’m using the cluster distance data from the University of Sheffield OAC datasite. To get a sense of fit I’m using entropy scores, given by the following equation:

Where pi is the distance of a given OA to a given supergroup cluster centre with respect to the other distance to centres. Essentially this is a measure of evenness, in terms of OAC we’d like the results to be less-even as this would suggest that one distance to centre is much smaller than the others indicating a good cluster assignment, OAs that are more-even are indicative of OAs which don’t fit as well into a single OAC class. In the map below a lower entropy score indicates less evenness and hence more a more discrete assignment of OAC class.

The pattern that seems to emerge is that urban areas, such as London, and extremely remote areas, such as the highlands of Scotland, do not fit the classification so well. I quickly tested this conclusion by summarising the entropy scores by the rural urban classification 2004 from the ONS.

This graph seems to confirm the visual reading of the map to some extent, the fit is worst for Urban areas, better for town and fringe, best for villages and slightly worse again for Hamlets and Isolated Dwellings. This graph was created only from data pertaining to OAs in England and Wales though, as Scotland has a different classification as is its want. The effect of Scottish OAs may lift the value for Hamlets though, as Scotland has more remote areas than England and Wales in general. I’ve also created a graph for the Rural and Urban Classification 2004 using the combination classification that takes into account ‘sparseness’ as well. Ostensibly sparcity relates to the number of housholds in the surrounding 30km of a grid which has been aggregated to OA level. From this a distinction of sparse and less sparse is created, I’ve got no idea what this means and it seems useless and confusing, however it does back up the earlier poitn for what it’s worth:

Areas that are ‘sparse’ seem to be less well classified than areas that are ‘less sparse’ – I’ve no idea what that means though. Nevertheless the pattern is much the same.

Essentially OAC works better if you’re not classifying extremely urban, or extremely rural areas. I think someone should look at the rural urban classification though, or a least write a sensible description of what is actually meant by sparse or less sparse – a less sparse urban areas? I wish I knew what that meant!

I have recently been made secretary of the Output Area Classification User Group (OACUG), a research group based at the Royal Statistical Society (RSS). This is a second role to complement my position as Postgraduate Representative of the Quantitative Methods Research Group (QMRG) at the Royal Geographical Society (RGS). My colleague Alex Singleton has also been made chair of the OACUG.

The Output Area Classification (OAC) is a geodemographic classification, meaning that it is a system for characterising the type of people that live in a given area based upon their demographic data from the 2001 census. There are 3 hierarchical levels to OAC, consisting of 7 groups at the top, 21 at the middle and 52 at the bottom level. The intent of OAC is to provide a simple and general measure of ‘neighbourhood type’. The scale at which OAC is reported is Output Area (OA) which is an areal unit that contains roughly 125 households.

A good visualisation of OAC exists at on the London Profiler site.

Further information at the OACUG and QMRG.

On friday (10/07) I attended a workshop on the usage of the Output Area Classification (OAC), aimed at local government and the public sector. I have experimented with a number of geodemographic classifcations, both commercial (experian’s mosaic, CACI’s acorn and health acorn) and non-commercial (OAC and Petersen et al’s LOAC (2007)) and was interested to see the experienced line up presenting.

Tim Allen of the LGA introduced why customer insight is increasingly important, in the public sector this translates to understanding the situation and needs of your constituents, and directing spending accordingly. This introduction of the potential of OAC in the public sector pushes the merits it holds to assess neighbourhoods, communities and places. This trinity was oft mentioned, however I am suspicious that they were being conflated to mean similar things when in fact OAC is only realistically a window into neighbourhood/areal characteristics.

Second up was Dan Vickers of Sheffield University, who developed the OAC for the Office for national statistics, who went onto describe some of the characteristics of OAC and the decisions that went into its construction.

The remaining sessions focused on how OAC can be used to examine dataset and find trends, Martin Callingham, visiting Professor at the University of London, showed its use in profiling populations, essentially tagging locational data with the appropriate group and comparing it to the national average. In doing this he gave a lot of examples, but went onto describe OAC as fundamentally being about ‘place’ something I’m not convinced is true.

Likewise John Fisher of local futures, described how OAC can tell “stories of Britain”, which brings the work of Doreen Massey to mind, the idea of place being the composite of “stories so far”, however this is a post-structuralist view of place-construction which itsn’t reflected in the way OAC is cast, with strongly defined boundaries, absolute assignments and statistical relevance. Fisher brings in further element of government into the OAC agenda, referencing ‘place-shaping’ and ‘total place’, sustainable communities and localism. All elements which OAC may have a role in, but a role that needs careful shapign and consideration not broad geodemographic strokes.

Keith Dugmore, of Demographic Decisions, was more measured in his talk, which was quite interesting and revealed a geodemographic approach to assessing sample surveys, this is achieved through OAC coded surveys such as the British Household Panel Survey, or the Expenditure and Food Survey and allow for small area estimates for local areas based on the results of these surveys. The key failing picked up was a lack of confidence intervals, but these are easily added in reality.

Michael Willmott wrapped things up with a forward looking commentary on the trajectory of OAC and social research, unfortunately given apparent inability to distinguish neighbourhoods, communities, and places in some of the talks, and amongst some of the participants, it may have been overly optimistic.

I do think that geodemographic classifications have a role to play, simply that there needs to be a greater understandng of how they are best interpretted.