In Finland, municipalities are incredibly powerful; like local authorities in the UK, municipalities are responsible for local administration, but they also levy an income tax and are responsible for providing most public services. Municipalities were founded on the assumption of equality, which forms the basis for the reform considerations currently ongoing in the Finnish government. The fact is, municipalities simply aren’t equal, they vary wildly in population size and area; numbering 342 they are of rough numerical equivalence to UK local authorities, despite Finland itself having around 1/11th of the UK’s population. The population skew is greatly emphasised by the presence of cities such as Helsinki which cross municipal boundaries, and more remote municipalities whose geographical extent was set out by horse-and-cart distance. It is therefore understandable that the Finnish government would be interested in the possibility of mergers to reduce the number of municipalities, and create a system in which muncipalities serve a similar number, or at least a threshold, population. It is believed that this would make administration more efficient, as services can be centralised to a greater degree, and small municipalities which already share services can formalise this.

Some mergers have already taken place, and there are governmental incentives for merging. However, with the specified model that municalities should be reformed such that they have a base population of 20-30,000 people, from the Association of Finnish Local and Regional Authorities (Kuntaliitto), we can apply automated zone design scenarios to test the ‘what if’ aspect of creating a new Finnish municipal system based on preserving different characteristics.

The zone design tool that I use to test a few basic scenarios is the regionalization library in pySAL, a spatial analysis module for Python. This implements the max-P algorithm for spatially contrained clustering subject to a similarity matrix, and a threshold value. I specified the research design so that I was testing for the optimal new aggregation of the pre-existing muncipalities and tested 3 different scenarios: 1) No similarity measure (all municipalities assumed equal, but for population) 2) Similarity based on municipal tax regime, and 3) Similarity based upon municipal tax regime and % non-finnish speakers, which accounts culturally for the Sami people of Lapland and ethnically Swedish Finns.

The regionalisation requires that you create a contiguity matrix for the zones, I arbitrarily chose the queen case, and added bespoke contiguity for Finnish islands based upon proximity, this is easy to do in GeoDa and it outputs a .gal file which you can read into python. Then all you really need to do is the following:

import pysal
import numpy as np #required as pysal uses numpy arrays

#Read in your population and similarity data in some way,
#I tend to create a python list from a csv.

#convert the population and similarity data into
#numpy arrays, from lists called pop and sim
pop = np.asarray(pop)
sim = np.asarray(sim)

# Read in your precomputed Weights matrix
w = pysal.open("...\\QueenWeights.gal").read()

# Create an (optional) array of 1s to represent equality
# (replace sim in Maxp function call)
nosim = np.ones((342,1))

# Run solutions for maxp algorithms with specific parameters
r= pysal.Maxp(w,sim,floor = 20000,floor_variable = pop,initial=100)

# Write r.regions to outfile to get regional assignments,
#this can be joined to shp in ArcGIS.

The maxP algorithm works by first randomly creating a set of possible zoning configurations, then it chooses the current optimum and seeks to refine it using a computationally expensive zone-swapping method. Optimality in this case is defined by minimising dissimilarity whilst obeying the threshold population constraint. The API reference for regionalisation in pysal is very good.

Here are some of the results I produced for this basic approach:

Zone Design - Pop > 20,000 with assumption of Municipal Equality

Zone Design - Pop > 20,000 with similarity of tax regime and language

Unlike some of the more advanced zone-design algorithms, pySAL doesn’t yet provide a way of preserving or optimising area shape characteristics, so you can get sliver-like polygons forming. Nonetheless it presents an interesting insight and a set of functions from which a more advanced/bespoke algorithm could be built.As it turns out the islands that I arbitrarily allocated to the contiguity matrix are actually quite a contentious topic and given their strategic significance to Sweden are in fact neutral territories which would be untouched by any redesign of municipal structure.

As ever, local knowledge is important, and for the economists at VATT this is a real task to undertake. They won’t be doing anything quite so crude, they do however have a curious spatial problem to deal with.

I though I would explore the possibility of creating thematic maps using Python, this post documents my initial attempt. The output is hence rather basic, but encouraging. The primary reason that I wanted to test the mapping potential of python is to allow for some basic automated map production in order to quickly visually assess the geographical patterns contained within large data sets. This is something that I am at a loss to do in ESRI’s ArcGIS, although that might change in ArcGIS 10. For fans of R I know it can be done there, however R is too tricky for me! My colleague James Cheshire explains the method in R here.

The first hurdle in map making is getting the data in, for this I used the shapefile reader that Zachary Forest Johnson put together for his excellent blog ‘IndieMaps.com‘. This allowed me read in any of my masses of pre-existing Shapefile format datafiles, and indeed use the python scripting functionality in ArcGIS to perform spatial operations and then output a map quickly and without the hassle of dealing with ArcGIS layouts.

Once you have download the shapefile reader, it is easily implemented using:

import shpUtils   #imports the shapefile reader
#Load a shapefile into an object called shpRecords
shpRecords = shpUtils.loadShapefile('\filename.shp')

This is undoubtedly simple, what you then have is a (slightly) complex object which contians all of the shapefile data nested as lists and dictionaries. In order to get my head round this I spent some time investigating it, a standard shapefile that contains areal geographies (i.e. UK Output Areas) will have a similar set up to this:

• The first list (shpRecords[i]) records the number of complete geometries, this corresponds to the number of rows in the attribute table. Thus a single polygon has 1 row in the attribute table and 1 list (list index 0) in Python.
• The second dictionary (shpRecords[i]['key']) records two branches, reporting either the ‘dbf_data’ from the attribute table, or the ‘shp_data’ from the .shp file describing the underlying geometry.
• Choosing the ‘dbf_data’ key (shpRecords[i]['dbf_data']) allows you to see the attributes recorded column-by-column for each row (and hence each geometry) in the attribute table. Thus shpRecords[i]['dbf_data']['name'] will return the attribute value for the field ‘name’ for the ith geometry in the shapefile.
• Choosing the ‘shp_data’ key (shpRecords[i]['shp_data']) allows you to access the various components of the shapefile’s geometry. In the case of a polyline/polygon you get dictionary items ‘ymax’, ‘ymin’, ‘xmax’, ‘xmin’, ‘numpoints’, ‘numparts’ and ‘parts’. Clearly the first 6 items are properties of the ith geometry you are querying, so it allows you to form a bounding box, get the number of vertices in the line/polygon, and draw separate lines/polygons if the shapefile is setup to have spatially discontinuous shapes for each row.
• The thing we are most interested in is the ‘parts’ dictionary key, as this contains all the coordinates for the particular geometry being considered, this is accessed as: shpRecords[i]['shp_data']['parts']. The next list (shpRecords[i]['shp_data']['parts'][j]) thus allows you to distinguish between parts in a multipart file. i.e. the jth part of the ith geometry.
• Having come this far, one final dictionary allows us to see the coordinates themselves, this dictionary simply offers us ‘x’ or ‘y’. Thus finding the x-coordinate of the ith geometry and jth part is accessed by: shpRecords[i]['shp_data']['parts'][j]['x'] – simple!

I have been using matplotlib – a python library for scientific visualisation a lot recent, and have found it a very simple and powerful resource, so I thought I’d see if it could be made to draw a map.

Firstly import the pyplot element which does all the figure drawing:

import matplotlib.pyplot as plt

Now lets use the “fill” component of matplotlib to draw all the geometries in a shapefile – my shapefile is Output Areas in Southwark. Firstly we need to loop through each geometry, and then draw a polygon using all the points contained within each geometry. I omitted a loop for multipart geometries as my shapefile has none, however this would be very easy if the data did have multiple parts- simply add a loop in the middle!

for i in range(0,len(shpRecords)):
 # x and y are empty lists to be populated with the coords of each geometry.
 x = []
 y = []
 for j in range(0,len(shpRecords[i]['shp_data']['parts'][0]['points'])):
  # This is the number of vertices in the ith geometry.
  # The parts list is [0] as it is singlepart.

  # get x and y coordinates.
  tempx = float(shpRecords[i]['shp_data']['parts'][0]['points'][j]['x'])
  tempy = float(shpRecords[i]['shp_data']['parts'][0]['points'][j]['y'])
  x.append(tempx)
  y.append(tempy) # Populate the lists  

 # Creates a polygon in matplotlib for each geometry in the shapefile
 plt.fill(x,y)

plt.axis('equal')
# This sets the x and y axes as equal intervals.
# NB this script will only work for projected data, for geographical
# coordinate systems get ready to do some maths  

plt.show() # Draws the map!

This is the simplest form of the script, it will simply draw the shapefile with each area filled a random colour. This is not that useful, but it is easy to create a thematic maps of categorical data, so let investigate a way of doing that. I’ve got data for the Output Area Classification, which is a clustering of areas by social characteristics, I know that there are 7 supergroups in the classification, named numerically, so before all the processing of the shapefile I can create a dictionary of colour choices for each group. I’m using hexadecimal colours that I got from Cynthia Brewer’s website for a ‘qualitative’ 7 class classification. The dictionary looks like this:

oacSGroups = {'1':'#A6761D','2':'#E6AB02','3':'#66A61E','4':'#E7298A',\
'5':'#7570B3','6':'#D95F02','7': '#1B9E77'}

Thus the key ’1′ returns the associated hex colour, this can be linked to the ‘dbf_data’ key in the shapefile. In the plt.fill() component I simply have to specify the colour choice, thus we alter the line in the above script to read:

plt.fill(x,y,fc = oacSGroups[str(int(shpRecords[i]['dbf_data']['supergroup']))]\
,ec = '0.7',lw=0.1)

‘fc’ is the ‘foreground colour’ we are asking python to make the colour equal to the value in the oacSGroups dictionary where the key is the value contained in the attribute table for the ith row in the ‘supergroup’ field. Thus if the ith row had a ‘supergroup’ value of ’7′ that foreground colour would be set to ‘#1B9E77′. ‘ec’ is ‘edge colour’ and ‘lw’ is linewidth, here I have set the values to display fine, light grey lines.

Finally, as basic a map as this will turn out to be, we wouldn’t be anywhere without a legend. The following a a very basic, wholy manual way to add a legend to the map:

p1 = plt.Rectangle((0, 0), 1, 1, fc="#A6761D")
p2 = plt.Rectangle((0, 0), 1, 1, fc="#E6AB02")
p3 = plt.Rectangle((0, 0), 1, 1, fc="#66A61E")
p4 = plt.Rectangle((0, 0), 1, 1, fc="#E7298A")
p5 = plt.Rectangle((0, 0), 1, 1, fc="#7570B3")
p6 = plt.Rectangle((0, 0), 1, 1, fc="#D95F02")
p7 = plt.Rectangle((0, 0), 1, 1, fc="#1B9E77")

plt.legend([p1,p2,p3,p4,p5,p6,p7], ["Super Group 1","Super Group 2",\
"Super Group 3","Super Group 4","Super Group 5","Super Group 6","Super Group 7"], loc = 4)

This simply creates 7 rectangular plots which don’t appear on the plotted output, but instead are passed to the legend creator, each rectangle has the appropriate colour to match the mapped representation, and a label, shown int he legend as two ordered lists. The ‘loc’ tag allows the setting of where the legend will appear, 4 denotes the bottom right corner. the tag ‘title’ allows you to add a title to the legend as a string.

An example output looks something like this:

This took a couple of seconds to produce, and accounts for 846 individual geometries, which actually have quite a number of vertices.

I’ll update the blog should I find new methods to visualise spatial data in python.

As an extension of the previous post, I created MDS mappings for Inner London using the OAC variables. I am using the ONS definition of Inner London, as opposed the commonly accepted definition, thus Inner London includes Haringey, Newham and the City, but excludes Greenwich.

The nature of MDS is to scale a pairwise distance matrix, thus the number of entries in the matrix increases as the square of the number of observations, rapidly becoming very large. There are 9759 OAs in Inner London, requiring a distance matrix of 95,238,081 cells. I would like to compute the MDS mappings for Greater London as well, however there are 24,140 OAs giving a distance matrix of 582,739,600 cells, that is over 1/2 billion (American bn) entries and cannot be dimensioned on 32-bit machine. I’ve looked into 64-bit Python, but have yet to find a solution that doesn’t completely fill my computers memory and create an enormous pagefile.

In the previous example I used canberra distance, but found that in the larger data space of inner London I was having issues with very small fractions skewing the outcome, so I calculated the distance matrix using Bray-Curtis distance using SciPy’s spatial.distance library. The formula for bray-curtis is as follows (you’ll note it is a data normalisation method too):

where k is the particular variable relating to the pair xi and xj and dij is the resultant distance matrix.

The mappings were, as before, produced in greyscale and RGB.

Greyscale MDS mapping of Inner London using 41 OAC Variables

RGB colour mapping for MDS of 41 OAC Variables

I feel that the representations produced are quite effetcive in demonstrating the wide mix of social environments in Inner London, the greyscale is particualrly good at picking up the acknowledged patterns of deprivation, particularly east/west  and north/south. The colour representation does this too, but with an additional layer of complexity that seems to give a more nuanced reading of socio stratification in Inner London, with specific colour groups apparently marking out notional neighbourhoods – complementing the reading of London as a ‘city of villages’.

Acknowledgement

Census data is Crown Copyright 2010 from CasWeb, boundaries are Crown Copyright 2010 from UKBorders, an Edina/JISC supplied service.

A little while ago I created some basic measures of spatial equity for my main study site in Southwark, London. Spatial equity in this case relates to a measure of the ‘fairness’ of spatial distribution of services. The NHS as a public institution has a requirement in its universal terms of service to provide a fair service to all.

The following maps aim to show how different areas in Southwark, in this case output areas (OAs), have different characterisitics in terms of: the level of primary care provision available, and the distance to centres of primary healthcare. Following Truelove (1993), Talen and Anselin (1998) and Ricketts et al (1994) the first 3 maps use a buffer-approach to spatial equity, whilst the final shows a gravity model approach.

Figure 1: Spatial equity measured with a 500m buffer around GPs

This first map (figure 1) demonstrates that large parts of Southwark do not have access to healthcare services within 500 metres (euclidian distance), whereas the best served areas have access to more than one GP surgery and as many as 24 individual doctors.

Figure 2: Spatial equity measured with a 750m buffer around GPs

Figure two demonstrates that with a 750m buffer most areas are served, although there are still unserved areas, particularly in the south of the borough. The most well-served areas not have access to as many as 48 doctors.

Figure 3: Spatial equity measured with a 1000m buffer around GPs

A 1km buffer still shows areas of Southwark which are unserved, particularly in the south. My recent working paper features a map which confirms that residents of these areas are less likely to use Southwark services than those in the more core areas in the centre of the borough.

Figure 4: Spatial Equity measured by Log of the Gravity Potential

This final map uses a distance decay function rather than a buffer to represent spatial equity and is specified thusly (Talen and Anselin, 1998 p.600):

where Sj is the size of a facility (measured by number of doctors, operating capacity etc.) at location j and d is a distance decay factor between area i and facility j with a friction parameter alpha, here set to 2.

The result is not hugely different to the buffered approaches, giving a similar account of affairs. It is notable that in all cases the spatial equity correlates with provision of social housing. In the UK context Southwark is a special case, being amongst the most deprived Local authorities by IMD07 rank, in which fair access to services is skewed towards the needs of the more deprived, whether or not uptake, or ability to uptake actually reflects this is another question.

References

]]> http://danieljlewis.org/2009/12/08/basic-equity-maps-for-southwark/feed/ 0 http://danieljlewis.org/2009/09/21/computers-and-the-renaissance-of-cartography/ http://danieljlewis.org/2009/09/21/computers-and-the-renaissance-of-cartography/#comments Mon, 21 Sep 2009 14:34:58 +0000 Daniel Lewis http://danieljlewis.org/?p=53

Dr. Muki Haklay directed me toward a pamphlet entitled “Computers and the Renaissance of Cartography” by T A Margerison of the Experimental Cartography Unit; it is a wonderful example of mapping, as well as of publication, no doubt something to do with the Royal College of Arts. Perhaps the most incredible aspect of the work is that the featured maps, which would be exceptional nowadays, were created in the late 70s. Don’t take my word for it however, have a look at these two examples from the pamphlet:

This extract is from a map of cobalt for a part of the Shetlands: quantities are recorded by the length of arm (Margerison, 1976 p.12)

This extract is from the 'Atlas of Global Seismology: 1909-1969' which is based on the large digital data bank of seismic records held by the Institute of Geological Sciences. The seismic information for each "square" is plotted directly by computer in bars of different thickness, oriented N.S. E-W and diagonally. A brown tint shows areas in which some activity has been recorded during the total period - e.g. on maps of other time periods (Margerison, 1976 p.11)

Reference:

Margerison, T A (1976) Computers and the Renaissance of Cartography. NERC Experimental Cartography Unit, RCA, London