I work a lot with flows and spatial interactions, one thing that I’ve wanted to do for a while is compute a mean flow direction surface. Unfortunately, arithmetic means don’t work for angular data, this is because it cannot account for the circular nature of the distribution of angular measurements. For instance the angles 5 degrees and 355 degrees are seperated only by 10 degrees, but their arithmetic mean is 180 degrees -w ay off, it should be 0 degrees!

Luckily, Brunsdon and Charlton have published on this very subject, so I took it upon myself to implement a weighted circular mean function in Python. The key obstacle was learning about complex numbers, about which, up until this point, I had no idea about at all!

The first thing to do is calculate the angle between a set of candidate points (such as people) and a set of services (such as Medical Centres). This is simple enough to do using, and would look something like:

import math

math.atan2((y2-y1),(x2-x1))

In which the pair (x1,y1) is the location of the candidate point, and (x2,y2) the location of the allocated service for that candidate point. The line linking these two points defines a flow from a candidate point, to a servcie and vice versa.

Having calculated all of the angles, I used ArcGIS to create an output grid, at the extent of the candidate points, using the “fishnet” function which creates a vector grid of prespecified dimensions.

The beauty of Brunsdon and Charlton’s method is that it uses a local method of approximation, this means that for each cell in the output grid, a mean direction can be calculated based upon the values of nearby points, applying a weighting allows for more distance points to have less of an effect on the mean direction.

Firstly, I read all the candidate points into a KDTree structure, this allows me to search for local points, at the same time I also create an array of the angles for those candidate points.

from scipy.spatial import cKDTree
import numpy as np

tree = cKDTree(treepoints)
res, idx = tree.query(testpoint,300000,0,2,100)
res = res[0][np.where(res[0] < np.Inf)[0]]
idx = idx[0][:len(res)]

The tree takes a numpy array of coordinate pairs, and the query method returns an array of distances to points (res) and their index value in the original array of coordinates (idx). The testpoint is a cell in the vector grid; 300000 is the k-number of nearest neighbours to find, here I have simply set it arbitrarily high in the context of my dataset; 0 is for approximate nearest neighbours, here I’ve specified exact; 2 indicates the use of euclidian distance; and 100 is the threshold, neighbours won’t be returned if they are further than 100 metres away. The penultimate line simply returns an array that is shortened to just those values which are less than 100m away (i.e. less than infinity) – points over 100m away are returned as value Inf.

The next step is to actually compute the mean direction, this requires a special approach using complex numbers however. Brunsdon and Charlton show that a direction can be stated as a complex number z in which z = exp(iθ) this is effectively: z = cos(θ) + i sin(θ)  in which i is an imaginary number. We can restate our directions in Python using:

import cmath

thetas = angles[idx]
cThetas = []
for i in xrange(0,len(thetas)):
    cThetas.append(complex(np.cos(thetas[i]),np.sin(thetas[i])))
cThetas = np.array(cThetas)

Here, the complex function allows the complex number representing an angle to be stored in a list, which I convert (lazily) to a numpy array. The first term, thetas, is using the idx array from the cKDTree to cleverly index the relevant angle records from the angles array which stores all the angle values in the order of entries for the cKDTree.

Next a temporary variable is created which calculates the mean direction:

temp = np.sum(cThetas)/np.absolute(np.sum(cThetas))
MeanDir = np.angle(temp, deg = True)

The mean direction is given by the argument (Arg) of the resultant complex number, Python implements this with the np.angle function, where deg = True returns the angle in degrees, and False in radians.

So far this is the unweighted mean, aggregating directional observations within a 100m disk (see also: uniform disk smoothing). To introduce weighting we must first define a weighting scheme, I’ve used the one suggested by Brunsdon and Charlton, which is Gaussian, and might look at bit like this:

def gaussW(dists,band):
    out = np.zeros(dists.shape)
    for i in xrange(0,len(out)):
        temp = np.power(dists[i],2)/(2.0*np.power(float(band),2))
        out[i] = np.exp(-1.0 * temp)
    return out

weight = gaussW(res,100)

Quite simply, I pass the distance array res to the gaussW function and it gives me back an array of weights for that ordering of distances. Using this I can redo the mean direction thus:

temp = np.sum(weight*cThetas)/np.absolute(np.sum(weight*cThetas))
MeanDir = np.angle(temp, deg = True)

There you have it! Attached is the script I used. Obviously, Brunsdon and Charlton implement a variance and a couple of visualisation devices, but these should be simple enough to implement now!

I created an output for flows of patients to GPs in Southwark, visualised using one of ESRI’s circular/direction colour ramps from colour ramp pack 2. Not sure how best to visualise the legend at this point though. NB. 90 is north, -90 is South, 0/-0 is East and 180/-180 is West. The map is visualised to show the 4 cardinal directions, but the output is in fact continuous.

My example script is here. Note that I am using dbfpy to read and write to shapefile DBF tables directly.

Using the excellent SANET extension for ArcGIS 9.3 I was able to take some of my data for Southwark that I had geocoded to address level, and estimate the population density using the OS Mastermap ITN product. The procedure is essentially a Kernel Density Estimation that takes place on a given network rather than across 2D space, this effectively controls for the effect of spatial structure, such as urban form, of which the data relates to residential locations. The estimation is made for c.300,000 people in Southwark on a network with around 30,000 road segments so it is to be expected that the calculation takes several hours to run. The KDE process is parameterised in much the same way as the straightforward density estimation procedures in the ARCGIS Spatial Analyst toolboxes, bandwidth and cell size are specified. In this case though cell size relates to the length of segments into which the network has to be cut in order to represent the output. Additionally, SANET allows you to control how you handle road intersections, either by using a continuous or discontinuous approach to the bifurcation, i arbitrarily chose the continuous approach, essentially meaning that the density estimation can turn corners. A straightforward representation can be made in 2D as below.

The interesting aspect to this image that is obscured in 2D smoothed representations is the relative usage of different streets, clearly visible are the residential streets as distinct from the more commercial area on Southwark’s Bankside, and along major roads, and the effect of open space and water features in reducing network density (i.e. if only one side of a road has residences on it). I’ve attempted to explore this further by using ArcScene’s 3D visualisation capabilities, but the complexity of the data make this an incredibly arduous process. The result i was able to obtain outside of ArcScene simply crashing are below.

In this example, Southwark is presented in a kind of 2.5D perspective in which the streets have been extruded so that their height represents the population density at that point. I’ve included some contextual elements, the Thames, and parks, wooded areas, and other water features. Whether or not this image is in anyway an improvement over a simple 2D representation is open to debate, but the selections below do present an interesting cross section of the data.

I saw a thought provoking presentation recently, given by Wenwen Li of the University of California Santa Barbara, the talk was a wide ranging insight into Cyber Infrastructure, its uses for geospatial information, and some of the computational techniques that underpinned the project. One element of the project involved zone design for the greater Los Angeles region, and involved the implementation of an algorithm that was intended to aggregate small areal units into larger zones whilst meeting a number of conditions, principle among these conditions was ‘compactness’. The output looked very much like a single hierarchy of Christaller hexagons, and this got me thinking about the nature of space and compactness.

Christaller’s hexagons are the defining illustration of something called ‘central place theory’, a geographical abstraction that idealises settlement pattern based upon an underlying space which is assumed to be isotropic. The assumption of spatial isotropy is the big leap in this model, it assumes that the ‘friction of distance’ from any given point increases at an equal rate whichever way you go from that point. Clearly such a suggestion is not applicable to Los Angeles, where huge freeways and interchanges can make adjacent parcels of land remote neighbours, and increase the connection between advantageously placed non-adjacent sites? Surely a city in which sprawl and ribbon development, as well as segregated communities should be modeled differently? Why then do many of our zoning algorithms favour compact ‘circular’ shapes, very much in the christaller mould, and why do we reject uncompact areal features as ugly slivers? In short, how did the circle come to be the ideal shape of zone in regional studies? Certainly, it is easier, both implementationally and conceptually, to model circles than to consider optimising a zone system over an n zone by n zone similarity matrix pertaining to variables which may be important to aggregating any set of areal units. However, as we explore more and more the complex systems defined by cities and regions, surely there is a need to start integrating a more realistic anisotropic view of space, one in which the friction of distance from any given point in any given direction is defined by the underlying demography, built environment and/or infrastructure.

One such attempt at this, AMOEBA (A Multidirectional Optimum Ecotope-Based Algorithm), developed by Aldstadt and Getis, is worth noting. In this algorithm, zones are defined via the Getis-Ord Gi* statistic, which is a local statistic for identifying clustering, thus zones are defined by local conditions, which are free to vary anistropically across space, rather than by a predefined preference for circles. Spectacularly this algorithm is implemented in the superb clusterpy python module for spatially constrained clustering.

I’ve been doing a bit of work recently on segregation with Pablo Mateos, and having gone through the motions with aspatial indices of segregation (the classics): dissimilarity, exposure and so on, I decided to investigate the more explicitly spatial ones. Taking a lead from Reardon and O’Sullivan’s (2004) paper “Measures of Spatial Segregation” in sociological methodology, I got in touch with David O’Sullivan and he, and his student Seong-Yun Hong, helped me with the implementation of some spatial measures of segregation. This post specifically concerns spatially weighted entropy – a measure of population diversity. Reardon and O’Sullivan define spatially weighted entropy as:

This equation describes the ‘entropy’, derived from Shannon’s information theory, for each grid cell in the image (below) in which each cell value results from the entropy computed for a 1km ‘neighbourhood’ p around each cell (essentially a circular buffer). The ethnic group in question is given by ‘m’ (with the pi representing the proportion of a given group in a given neighbourhood) and relates to ethnic groups defined from the Southwark patient register using Onomap, the groups defined are: African, East Asian and Pacific, European, Muslim, South Asian, British, Eastern European, Hispanic, and Unclassified or Other. The Onomap software is able to apply this classification by looking at the forename and surname combination of patients registered to use Southwark GPs, or patients living in Southwark but using GPs outside of Southwark. The cells in the image relate directly to the residential locations of patients, who were geocoded to their household using the Ordnance Survey’s Address Layer 2 product, therefore empty cells are areas within which no recorded patients were found, such as parks, and transport infrastructure. As the data underlying this is from patient registrations with GPs, we have to accept that the data is likely to be partial, with potentially systematic biases in those people who have registered – young men and people from countries where GPs as a method of primary care do not exist- may have been omitted.

In the image, higher values of entropy indicate greater diversity of population by ethnic group, the resultant images is unsurprising in terms of Southwark, with the Dulwich Village area showing as the least diverse place, home as it is to more affluent, generally ‘British’ groups. Likewise historical factors regarding access to housing have shaped the lower entropy scores in the middle of the borough – home to African populations and the North East, home to the British working classes who were rehoused from the now more African areas in the middle of the borough. Finally, the greater Waterloo- Elephant and Castle region in the north-west shows up as the ethnic melting pot in the borough.

In the image above, the 1km neighbourhood defined in the spatially weighted entropy score has a smoothing effect, I experimented with smaller values for the neighburhood size, and found that the resultant output did not change dramatically from that obtained above. At the end of the day, the selection of neighbourhood size is largely arbitrary and will depend on sociocultural factors of the area and it’s people. Similarly, as there is no data for the regions outside of Southwark we are more uncertain of the values at the edges than in the middle of the borough as we are only sampling from within Southwark itself. Nonetheless, this representation of Southwark goes somewhat beyond what is possible using the commonly used output zones defined by the census.

In Finland, municipalities are incredibly powerful; like local authorities in the UK, municipalities are responsible for local administration, but they also levy an income tax and are responsible for providing most public services. Municipalities were founded on the assumption of equality, which forms the basis for the reform considerations currently ongoing in the Finnish government. The fact is, municipalities simply aren’t equal, they vary wildly in population size and area; numbering 342 they are of rough numerical equivalence to UK local authorities, despite Finland itself having around 1/11th of the UK’s population. The population skew is greatly emphasised by the presence of cities such as Helsinki which cross municipal boundaries, and more remote municipalities whose geographical extent was set out by horse-and-cart distance. It is therefore understandable that the Finnish government would be interested in the possibility of mergers to reduce the number of municipalities, and create a system in which muncipalities serve a similar number, or at least a threshold, population. It is believed that this would make administration more efficient, as services can be centralised to a greater degree, and small municipalities which already share services can formalise this.

Some mergers have already taken place, and there are governmental incentives for merging. However, with the specified model that municalities should be reformed such that they have a base population of 20-30,000 people, from the Association of Finnish Local and Regional Authorities (Kuntaliitto), we can apply automated zone design scenarios to test the ‘what if’ aspect of creating a new Finnish municipal system based on preserving different characteristics.

The zone design tool that I use to test a few basic scenarios is the regionalization library in pySAL, a spatial analysis module for Python. This implements the max-P algorithm for spatially contrained clustering subject to a similarity matrix, and a threshold value. I specified the research design so that I was testing for the optimal new aggregation of the pre-existing muncipalities and tested 3 different scenarios: 1) No similarity measure (all municipalities assumed equal, but for population) 2) Similarity based on municipal tax regime, and 3) Similarity based upon municipal tax regime and % non-finnish speakers, which accounts culturally for the Sami people of Lapland and ethnically Swedish Finns.

The regionalisation requires that you create a contiguity matrix for the zones, I arbitrarily chose the queen case, and added bespoke contiguity for Finnish islands based upon proximity, this is easy to do in GeoDa and it outputs a .gal file which you can read into python. Then all you really need to do is the following:

import pysal
import numpy as np #required as pysal uses numpy arrays

#Read in your population and similarity data in some way,
#I tend to create a python list from a csv.

#convert the population and similarity data into
#numpy arrays, from lists called pop and sim
pop = np.asarray(pop)
sim = np.asarray(sim)

# Read in your precomputed Weights matrix
w = pysal.open("...\\QueenWeights.gal").read()

# Create an (optional) array of 1s to represent equality
# (replace sim in Maxp function call)
nosim = np.ones((342,1))

# Run solutions for maxp algorithms with specific parameters
r= pysal.Maxp(w,sim,floor = 20000,floor_variable = pop,initial=100)

# Write r.regions to outfile to get regional assignments,
#this can be joined to shp in ArcGIS.

The maxP algorithm works by first randomly creating a set of possible zoning configurations, then it chooses the current optimum and seeks to refine it using a computationally expensive zone-swapping method. Optimality in this case is defined by minimising dissimilarity whilst obeying the threshold population constraint. The API reference for regionalisation in pysal is very good.

Here are some of the results I produced for this basic approach:

Zone Design - Pop > 20,000 with assumption of Municipal Equality

Zone Design - Pop > 20,000 with similarity of tax regime and language

Unlike some of the more advanced zone-design algorithms, pySAL doesn’t yet provide a way of preserving or optimising area shape characteristics, so you can get sliver-like polygons forming. Nonetheless it presents an interesting insight and a set of functions from which a more advanced/bespoke algorithm could be built.As it turns out the islands that I arbitrarily allocated to the contiguity matrix are actually quite a contentious topic and given their strategic significance to Sweden are in fact neutral territories which would be untouched by any redesign of municipal structure.

As ever, local knowledge is important, and for the economists at VATT this is a real task to undertake. They won’t be doing anything quite so crude, they do however have a curious spatial problem to deal with.

In some of the work I’m currently doing looking at households as derived from the Southwark patient register I wanted to go beyond a quantification of how many people lived in a households – a rudimentary household size, to looking at the composition of a household and hence what type of household it represented. In order to do this I looked at how types of household were generally reported in the UK Census, in European statistics, and in terms of social research on the life course, as well as in health literature itself. In terms of defining households, I found that although complex household typologies do exist, there exists a general set of likely household forms: as expected these revolve around the single, co-habiting, family, single parenthood, extended family etc models. As I have data on individuals I first decided to classify individuals into 5 broad categories that seem important in the literature and then look at the composition of these categories within households. The categories were:

1) Dependent Children (<18 yrs old)

2) Adult Male (18-65 yrs old)

3) Adult Female (18-60 yrs old)

4) Male Pensioner (65+ yrs old)

5) Female Pensioner (60+ yrs old)

Evidence suggests that these represent the coarsest categories that could usefully represent significant periods within the life course, as well as being relevant to changes in health status. In a sense, different type of household structure can be described by different combinations of these person classes for different household sizes.

I decided to test this by calculating all the possible combinations of these 5 classes for a 2 person household and then looking at their uptake in the actual household data I had derived from the Southwark patient register. It turned out that for a two person household there were 15 different ways in which you could combine the 5 person classes in order to create a unique household:

Children Only (Parents Unregistered); Single Parent Male and Child; Co-Habiting Men; Single Parent Female and Child; Single Parent Male Pensioner and Child; Co-Habiting Man and Woman; Co-Habiting Man and Male Pensioner; Co-Habiting Women; Single Parent Female Pensioner and Child; Cohabiting Woman and Male Pensioner; Cohabiting Man and Female Pensioner; Cohabiting Male Pensioners; Cohabiting Woman and Female Pensioner; Cohabiting Male and Female Pensioner; Cohabiting Female Pensioners.

Using this typology of 15 possible household types, I extracted the two person households from the larger dataset and wrote a Python script to classify these households. The result for 27,124 households was a follows:

What this graph seems to demonstrate is that roughly half of all 2 person households consist of a man and a woman (either adult or pensioner) cohabiting, and roughly a further 22% of same sex cohabitation. In this dataset for two person household, single parents only make up around 15% of households of which almost 13% is a single female parent (adult or pensioner) and a child. All other groups only make up around 13% of households, but crucially the only category in which no households were found to exist was the adult man cohabiting with a male pensioner category. Indeed many of the smaller categories can be interpreted as having inherently important social roles, the adult woman looking after a male or female pensioner for instance.

Essentially, the terrain of household type was a lot more nuanced and tricky than I’d at first though, made even more so by my realisation that as household size increases, the number of possible combinations of the person types within a  household increases dramatically. I wrote a python script to calculate the number of possible different sets of people for the household sizes 1 to 10:

This presents a difficult situation, even for reasonably small households. This is a problem known as “combinatorial mathematics” or “combinatorics“. I decided to see what I could learn about this distribution, so I looked for patterns in the sequence, as you are taught in pre-GCSE maths and soon found that the sequence had a constant fourth difference:

This constant fourth difference indicated that the sequence can be explained by a quartic function, of which is was easy to then calculate the form:

Sadly not one of those classically beautiful equations.

This all leads to the issue of how I now classify households, clearly the number of possible sets makes anything above around 4 people per household fairly intractable. I’ll experiment with 3 households and see whether I can account for most household types with a few set patterns and then look at households that fall outside of this remit.

Interesting none the less, I hadn’t expected to be doing much of this kind of maths!

I noticed in a beta of ArcGIS 10 (then called 9.4) that there was a ‘new’ option for computing a Geometric Median which didn’t exist in my copy of ArcGIS 9.3. This is an interesting concept, as in 1d statistics, the geometric (2d) mean is easy to calculate, being the average of all the X coords and all the Y coords. From stats we know that the Mean and Median value of a distribution will coincide if the data is perfectly normally distributed; however in the real world data usually will only approximate a normal distribution, leading to a mean value that is different from the midpoint, or median.  Therefore for a skewed distribution on the plane, we encounter a situation in which the mean is not necessarily the best representation of the ‘centre’ of the data, thus we may wish to calculate the median; doing so will also give us a good idea of the direction of the skew of the point pattern we are investigating. In calculating the median of a 2d point pattern we can express the problem as a need to:

minimise the sum of squared distances from all points in a distribution to a centre.

Thus it is reasonably clear that we are dealing with an ‘optimisation problem’, something that I have experimented with before in work I conducted using the ‘transportation problem’, a classic linear programming problem.

In terms of application, I though that finding the median of a distribution of people around a service would be a useful, albeit basic, indication of whether all people were making a similar trip to a service, or whether there were other effects at work (this would be evidenced by a median centre that was not close to the actual service location). I though I would be able to code the optimisation routine in Python using pre-existing insight. Notably, the wikipedia page on this details the Weiszfeld Algorithm as the acknowledged computational solution to the geometric median problem, it takes the form:

However, actually writing the algorithm proved somewhat tough. Essentially the answer is to start with a candidate data point (I started with the mean centre) and calculate the objective function – in this case the sum of the euclidian distances of all points from the candidate centre. Then pass the candidate point through the Weiszfeld Algortihm and reassess the objective function, at such a point as the objective function converges a median has been found. There is no guarantee that the median found is the optimal median though, and depending of the data there may be more than 1 optimal solution. Below is a solution for some of my data (the data has been randomly offset by 75m to preserve anonymity) on patient registrations to a doctor.

Here we can see that the mean and median centres are slightly different, suggesting that the patient population is skewed slightly northwards, most likely as a result of discontinuous urban infrastructure.

The scatterplot was achieved using the matplotlib Python plotting library. This was just a test, but I imagine more complex outputs can be achieved reasonably easily.

Notably, this technique is using euclidian distance, which in a dense urban environment may be misleading, I note that there is a relatively simple execution of the Dijkstra algorithm for shortest paths in Python, and I am curious whether this could be integrated to find a geometric median on the network, although I suspect that it may be unworkable due to computational time constraints, although for smaller problems it might be ok.

Naturally there are algorithms that can calculate a solution to the above for p-medians (i.e. several service centres in a population- commonly known as location-allocation), it is something that Paul Densham at UCL has worked on, and his code is making a return to service in ArcGIS version 10. I’m looking forward to seeing it, as it is a very difficult problem to solve (and in fact already has been ‘solved’), and not one I intend to investigate!

My code for the geometric median is here.

The Jenks Optimal, or Jenks’ Natural Breaks, Algorithm is a common method for classifying data presented in a choropleth map. It aims to present a series of break values that best represent the actual breaks observed in the data as opposed to some arbitrary classificatory scheme (i.e. equal interval), in this way the actual clustering of data values is preserved (subject to the arbitrary specification of k classes). The method was originally published in George Jenks’ (1977) Optimal Data Classification for Choropleth Maps and reportedly represented the culmination of 15 years research on the topic, the method primarily derived from Walter Fisher’s work ‘On grouping for maximum homogeneity‘. The specifics of the algorithm aim to create k classes so that the variance within groups is minimised, as such it is a problem of numerical optimisation.

A paper by Michael Coulson (1987) entitled In The Matter Of Class Intervals For Choropleth Maps: With Particular Reference To The Work Of George F Jenks details a method that Jenks apparently authored, but never published, to derive how optimum the number of classes chosen was, the method Goodness of Variance Fit (GVF) works by taking the difference between the squared deviations from the array mean (SDAM) and the squared deviations from the class means (SDCM), and dividing by the SDAM. Thus:

GVF = (SDAM – SDCM)/SDAM

However, it is likely this was never published as the GVF improves as the number of classes increases, until at such a points as there are the same number of classes as data points, the GVF reaches unity. Nonetheless, I have included a rudimentary example for calculating this statistic. In reality, this method is used to generalise data into a few classes for visualisation, so you are unlikely to be using more than 7 (+/- 2) classes; number of classes can be loosely assigned by looking at the distribution histogram, but often this is difficult.

The script is here.

Acknowledgement: The initial script I used for the Python conversion can be found (in JAVA and Fortran) here: https://stat.ethz.ch/pipermail/r-sig-geo/2006-March/000811.html

]]> http://danieljlewis.org/2010/06/07/jenks-natural-breaks-algorithm-in-python/feed/ 5 http://danieljlewis.org/2010/05/11/mds-for-inner-london/ http://danieljlewis.org/2010/05/11/mds-for-inner-london/#comments Tue, 11 May 2010 15:07:59 +0000 Daniel Lewis http://danieljlewis.org/?p=299

As an extension of the previous post, I created MDS mappings for Inner London using the OAC variables. I am using the ONS definition of Inner London, as opposed the commonly accepted definition, thus Inner London includes Haringey, Newham and the City, but excludes Greenwich.

The nature of MDS is to scale a pairwise distance matrix, thus the number of entries in the matrix increases as the square of the number of observations, rapidly becoming very large. There are 9759 OAs in Inner London, requiring a distance matrix of 95,238,081 cells. I would like to compute the MDS mappings for Greater London as well, however there are 24,140 OAs giving a distance matrix of 582,739,600 cells, that is over 1/2 billion (American bn) entries and cannot be dimensioned on 32-bit machine. I’ve looked into 64-bit Python, but have yet to find a solution that doesn’t completely fill my computers memory and create an enormous pagefile.

In the previous example I used canberra distance, but found that in the larger data space of inner London I was having issues with very small fractions skewing the outcome, so I calculated the distance matrix using Bray-Curtis distance using SciPy’s spatial.distance library. The formula for bray-curtis is as follows (you’ll note it is a data normalisation method too):

where k is the particular variable relating to the pair xi and xj and dij is the resultant distance matrix.

The mappings were, as before, produced in greyscale and RGB.

Greyscale MDS mapping of Inner London using 41 OAC Variables

RGB colour mapping for MDS of 41 OAC Variables

I feel that the representations produced are quite effetcive in demonstrating the wide mix of social environments in Inner London, the greyscale is particualrly good at picking up the acknowledged patterns of deprivation, particularly east/west  and north/south. The colour representation does this too, but with an additional layer of complexity that seems to give a more nuanced reading of socio stratification in Inner London, with specific colour groups apparently marking out notional neighbourhoods – complementing the reading of London as a ‘city of villages’.

Acknowledgement

Census data is Crown Copyright 2010 from CasWeb, boundaries are Crown Copyright 2010 from UKBorders, an Edina/JISC supplied service.

I was caused to consider the problem of generalisable human behaviour by a presentation on evacuation modelling. In my eyes the ability to model or predict anything, and make the implementation transferable across different contexts is contingent on the assumption that you know how people will act. The fallacy here is that you know how people will act, but you don’t necessarily know how individuals will act. It is easy (relatively) to aggregate across social characteristics and say: young people move at x speed, but old people move at y speed, and use it as a way of building socially stratified and perhaps logically more realistic models (this is often known as disaggregation). This approach seems to work well for regional systems, and large groups of people, however as the system of interest becomes more and more localised, individuals whose observable characteristics diverge enough from a generalisable norm can actually have an important role in the outcome of the model. I tend to think of this group as the 10% who change everything, although the actually percentage is likely to vary contextually.

In my work I have been looking at uptake and registration with GPs (doctors’ surgeries) with a view to isolating the demographic qualifiers of choice that create these different spatial patternings of uptake. Essentially attempting to find interesting social disaggregations within the data. In this work, which is at the level of the surgery, but for a PCT system (i.e. administrative health area), it is clear that some people, a small minority, act unexpectedly and differently to others with the exact same socio-economic characteristics, and that the problem is exacerbated when you look at small and smaller problems. For example, the largest surgeries in Southwark have between 10,000 and 20,000 registered patients, for these it is far easier to model patient registration as a function of distance than it is for a surgery of only 2,000 people. This is because larger surgeries can aggregate out the small number of ‘deviants’ better than a small surgery can. This is a defacto small numbers problem – the effect of a small number of outlying cases has a larger effect on smaller units of aggregation (surgeries) because they make up a larger proportion of the total population.

What does this mean for small-scale agent based simulations then? Well, as far as I can see it is very difficult to predict who out of a population is likely to diverge from their socially-stratified peers and be the outlying individual, and since the scale of simulation is so localised this uncertainty is liable to dramatically change the predicted outcomes. Thus in any case estimating the likely outcome within a margin for error is plausible – x people of y population were subject to some disadvantageous outcome (i.e. death/injury), but assessing where deaths/injuries occured, or the characteristics of who died/was injured may be a bit ambitious, or open to a quantitatively unacceptable uncertainty.

Naturally, understanding local level social systems should be a priority, and deriving generalisable rule-bases to give the best possible outcome for the incidence of a given phenomena is important. However, I think we always have to accept that in these circumstances a small number of people can have a large effect on the outcome in a way which is largely incalculable.