I work a lot with flows and spatial interactions, one thing that I’ve wanted to do for a while is compute a mean flow direction surface. Unfortunately, arithmetic means don’t work for angular data, this is because it cannot account for the circular nature of the distribution of angular measurements. For instance the angles 5 degrees and 355 degrees are seperated only by 10 degrees, but their arithmetic mean is 180 degrees -w ay off, it should be 0 degrees!

Luckily, Brunsdon and Charlton have published on this very subject, so I took it upon myself to implement a weighted circular mean function in Python. The key obstacle was learning about complex numbers, about which, up until this point, I had no idea about at all!

The first thing to do is calculate the angle between a set of candidate points (such as people) and a set of services (such as Medical Centres). This is simple enough to do using, and would look something like:

import math

math.atan2((y2-y1),(x2-x1))

In which the pair (x1,y1) is the location of the candidate point, and (x2,y2) the location of the allocated service for that candidate point. The line linking these two points defines a flow from a candidate point, to a servcie and vice versa.

Having calculated all of the angles, I used ArcGIS to create an output grid, at the extent of the candidate points, using the “fishnet” function which creates a vector grid of prespecified dimensions.

The beauty of Brunsdon and Charlton’s method is that it uses a local method of approximation, this means that for each cell in the output grid, a mean direction can be calculated based upon the values of nearby points, applying a weighting allows for more distance points to have less of an effect on the mean direction.

Firstly, I read all the candidate points into a KDTree structure, this allows me to search for local points, at the same time I also create an array of the angles for those candidate points.

from scipy.spatial import cKDTree
import numpy as np

tree = cKDTree(treepoints)
res, idx = tree.query(testpoint,300000,0,2,100)
res = res[0][np.where(res[0] < np.Inf)[0]]
idx = idx[0][:len(res)]

The tree takes a numpy array of coordinate pairs, and the query method returns an array of distances to points (res) and their index value in the original array of coordinates (idx). The testpoint is a cell in the vector grid; 300000 is the k-number of nearest neighbours to find, here I have simply set it arbitrarily high in the context of my dataset; 0 is for approximate nearest neighbours, here I’ve specified exact; 2 indicates the use of euclidian distance; and 100 is the threshold, neighbours won’t be returned if they are further than 100 metres away. The penultimate line simply returns an array that is shortened to just those values which are less than 100m away (i.e. less than infinity) – points over 100m away are returned as value Inf.

The next step is to actually compute the mean direction, this requires a special approach using complex numbers however. Brunsdon and Charlton show that a direction can be stated as a complex number z in which z = exp(iθ) this is effectively: z = cos(θ) + i sin(θ)  in which i is an imaginary number. We can restate our directions in Python using:

import cmath

thetas = angles[idx]
cThetas = []
for i in xrange(0,len(thetas)):
    cThetas.append(complex(np.cos(thetas[i]),np.sin(thetas[i])))
cThetas = np.array(cThetas)

Here, the complex function allows the complex number representing an angle to be stored in a list, which I convert (lazily) to a numpy array. The first term, thetas, is using the idx array from the cKDTree to cleverly index the relevant angle records from the angles array which stores all the angle values in the order of entries for the cKDTree.

Next a temporary variable is created which calculates the mean direction:

temp = np.sum(cThetas)/np.absolute(np.sum(cThetas))
MeanDir = np.angle(temp, deg = True)

The mean direction is given by the argument (Arg) of the resultant complex number, Python implements this with the np.angle function, where deg = True returns the angle in degrees, and False in radians.

So far this is the unweighted mean, aggregating directional observations within a 100m disk (see also: uniform disk smoothing). To introduce weighting we must first define a weighting scheme, I’ve used the one suggested by Brunsdon and Charlton, which is Gaussian, and might look at bit like this:

def gaussW(dists,band):
    out = np.zeros(dists.shape)
    for i in xrange(0,len(out)):
        temp = np.power(dists[i],2)/(2.0*np.power(float(band),2))
        out[i] = np.exp(-1.0 * temp)
    return out

weight = gaussW(res,100)

Quite simply, I pass the distance array res to the gaussW function and it gives me back an array of weights for that ordering of distances. Using this I can redo the mean direction thus:

temp = np.sum(weight*cThetas)/np.absolute(np.sum(weight*cThetas))
MeanDir = np.angle(temp, deg = True)

There you have it! Attached is the script I used. Obviously, Brunsdon and Charlton implement a variance and a couple of visualisation devices, but these should be simple enough to implement now!

I created an output for flows of patients to GPs in Southwark, visualised using one of ESRI’s circular/direction colour ramps from colour ramp pack 2. Not sure how best to visualise the legend at this point though. NB. 90 is north, -90 is South, 0/-0 is East and 180/-180 is West. The map is visualised to show the 4 cardinal directions, but the output is in fact continuous.

My example script is here. Note that I am using dbfpy to read and write to shapefile DBF tables directly.

The intent of this post is not simply to uncover where the highest density of people belonging to a particular ethnic group are, but rather to use the ‘location quotient’ (LQ) technique to compare the ethnic density in any one area to the overall ethnic density in Southwark, thus providing a relative insight into where the density of particular groups is more, less or as dense as expected.

Location Quotients tend to work with areal units, characterising different areas subject to a larger region and providing a basic insight into where functions are clustered. Because the Southwark patient register data is address geocoded, we would be losing some spatial information if we choose to aggregate the data, not to mention the question of which areal aggregation is best. More info on how to create location quotients here.

A Location Quotient has 3 possible interpretations; if it is around 1 then the ethnic population in that area is at the level we would expect given what we observe nationally. If the LQ is less than 1 then that area has a lesser population of a particular ethnic group that what we would expect based upon national figures. Finally, in the LQ value is over 1 this suggests a concentration of the ethnic group in the area which is greater than we would expect given nationally observed levels. A LQ is quite simply a rate-ratio.

Instead of the standrad areal approach, the maps here use a density estimation approach in which disaggregate point data is transformed into a representation of the continuous density function of the point distribution. The LQ can then be computed for each cell based on the density of that cell with respect to the total density of the surface. This creates a smoothed LQ surface which is readily interpretable in the same manner as above. The Kernel Density Estimation used to create the ethnic and total population density surfaces should be parameterised in the same way; these examples use a 250m bandwidth and a 25m cells size, which is largely empirically redundant, based on the input dataset’s spatial resolution, but creates a more aesthetically appealing mapped representation. Naturally, the procedure works well for clustered data, in Southwarks case for the African and Muslim groups.

I’ve been doing a bit of work recently on segregation with Pablo Mateos, and having gone through the motions with aspatial indices of segregation (the classics): dissimilarity, exposure and so on, I decided to investigate the more explicitly spatial ones. Taking a lead from Reardon and O’Sullivan’s (2004) paper “Measures of Spatial Segregation” in sociological methodology, I got in touch with David O’Sullivan and he, and his student Seong-Yun Hong, helped me with the implementation of some spatial measures of segregation. This post specifically concerns spatially weighted entropy – a measure of population diversity. Reardon and O’Sullivan define spatially weighted entropy as:

This equation describes the ‘entropy’, derived from Shannon’s information theory, for each grid cell in the image (below) in which each cell value results from the entropy computed for a 1km ‘neighbourhood’ p around each cell (essentially a circular buffer). The ethnic group in question is given by ‘m’ (with the pi representing the proportion of a given group in a given neighbourhood) and relates to ethnic groups defined from the Southwark patient register using Onomap, the groups defined are: African, East Asian and Pacific, European, Muslim, South Asian, British, Eastern European, Hispanic, and Unclassified or Other. The Onomap software is able to apply this classification by looking at the forename and surname combination of patients registered to use Southwark GPs, or patients living in Southwark but using GPs outside of Southwark. The cells in the image relate directly to the residential locations of patients, who were geocoded to their household using the Ordnance Survey’s Address Layer 2 product, therefore empty cells are areas within which no recorded patients were found, such as parks, and transport infrastructure. As the data underlying this is from patient registrations with GPs, we have to accept that the data is likely to be partial, with potentially systematic biases in those people who have registered – young men and people from countries where GPs as a method of primary care do not exist- may have been omitted.

In the image, higher values of entropy indicate greater diversity of population by ethnic group, the resultant images is unsurprising in terms of Southwark, with the Dulwich Village area showing as the least diverse place, home as it is to more affluent, generally ‘British’ groups. Likewise historical factors regarding access to housing have shaped the lower entropy scores in the middle of the borough – home to African populations and the North East, home to the British working classes who were rehoused from the now more African areas in the middle of the borough. Finally, the greater Waterloo- Elephant and Castle region in the north-west shows up as the ethnic melting pot in the borough.

In the image above, the 1km neighbourhood defined in the spatially weighted entropy score has a smoothing effect, I experimented with smaller values for the neighburhood size, and found that the resultant output did not change dramatically from that obtained above. At the end of the day, the selection of neighbourhood size is largely arbitrary and will depend on sociocultural factors of the area and it’s people. Similarly, as there is no data for the regions outside of Southwark we are more uncertain of the values at the edges than in the middle of the borough as we are only sampling from within Southwark itself. Nonetheless, this representation of Southwark goes somewhat beyond what is possible using the commonly used output zones defined by the census.

Amongst other things, I’m beginning to tap into a data source I have acquired for my research known as Hospital Episode Statistics (HES). These are datasets which record the particulars of hospital service by patients. Generally they have a bit of a learning curve, and require the gathering of several additional datasets in order to make them useful. Having gathered all this data and put in all within a MySQL database I decided to conduct a basic analysis, using my study site of Southwark as a guinea pig. Essentially I wanted to known whether more people from Southwark were using hospitals of outpatient appointments than we would expect from national (England) figures. There are many reasons why any given area might be using health care services at a greater or lesser rate than other areas, but for the moment I simply wanted to see whether there was any interesting patterns.

In the HES data it is simple to calculate the total number of people using outpatient care, what is more complex is deriving an expected score from the national data. I went about it in the following way:

Firstly, I took the ONS experimental population projections from mid-2008 and calculated the number of people in each Southwark LSOA, and at the national (England) level, for each of the available age bands by men and women. The population projection age bands are quite coarse, giving totals for 5 population groups: 0-15, 16-29, 30-44, 45-64 (for men) or 45-59 (for women) and 65+ (for men) and 60+ for women. This isn’t ideal, but the age bands do roughly correlate with the different groups of mortality causes in the Grim Reaper’s road map (Shaw, Thomas, Smith and Dorling, 2008). Then I calculated the admission totals for all of the age-sex bands nationally (England), with this I could create a ratio of admissions against popualtion nationally. By applying this ratio to the Southwark LSOA population projects I could create an expected value for number of admissions per areas. Finally it is simply a case of dividing the observed admissions by the expected and multipling by 100 to get a score.

I mapped the results as follows, a score of 100 suggests that the area is not different from the national picture, whereas a value higher than 100 suggests that the area has more people using hospitals than we would expect and a value lower than 100 suggests the converse.

In the case of Southwark, the pattern seems to follow those that are often observed in my work on Southwark, in that the Bankside areas, and the southern part of the borough, in addition with the north-eastern former docklands area have levels of admissions that are equivilant too, or lower than what we would expect nationally, whereas the central areas have admission numbers higher than the national level.

I’ve been doing some more analysis on the Southwark GP patient register at the household level. After a fair amount of cleaning and interpretation I’ve arrived at the following distribution of households.

There are a number of interesting things to say about this data, not least in the section that I’ve marked ‘larger social groupings’ as it seems to suggest a possible migrant social network effect, as the larger household groupings tend to be of minority ethnic groups, including Nigerians and other Africans, Hispanics and South-East Asians who are perhaps using cross-country social ties as help in getting established when first arriving in the UK. However, visually the shape of the distribution of household occupancy is very distinctive, and actually is very close to an exponential. Here I’ve taken the log of frequency of occurence and plotted the best-fit line through the plot:

This linear trend means that the model log(y) = -0.1635x + 4.602 is a good predictor of the number of Households we can expect to exist in Southwark for a given value of x, or occupancy.

It is not entirely clear however why this situation is the case. Firstly, it may just be an artifact of the data, either of the matching process that has occured between the patient register and OS AddressLayer2, the way that GPs encode patient addresses in the first place, or the fact that the patient register is only a sample of the total population of Southwark, i.e. those people who register with a doctor. Secondly, it may simply be a reflection of the structure of the built environment in Southwark – i.e. what kind of housing is actually available. However, the distribution is also subject to the choices of individuals or groups.

Currently, I am in the process of dissagregating the above characteristics and looking at trends by different population groups.

Since coming to government, new Conservative Health Secretary Andrew Lansley has sought to fulfil the pledge he made to put an end to local restructurings of NHS service delivery by authorities higher up the NHS hierarchy. Ostensibly he believes that local decision-making will have a better overall effect on the quality of outcomes for patients and hence lead to a better health service. Specifically he wants to provide GPs with an opportunity to work with community leaders and their local authorities to steer local services. The core elements actually do not differ greatly from the outgoing Labour policies, particularly with respect to patient choice; however I will argue that there is a clear danger in engaging to too great an extent with a purely ‘local’ approach, in general there seems to be something of a misconception in Government, particularly in the provision of local services (i.e. schools), that local approaches are somehow ‘better’.

Firstly, let us consider something that the Government seems to do without fail, something that I, as a Geographer, find to be a grave sin of omission. That is the apparently indiscriminate use of spatial qualifiers without so much of an explanation as to their meaning. The use of ‘local’ and ‘community’ are spectacularly misleading without qualification, and yet they are often used because people seems to think they understand what is meant by them – everyone considers themselves part of a community, and local to a service – but will these personal feelings about their socio-spatial connections actually translate to the ability to input on healthcare decision making? My investigation of access and registeration of patients to GPs in Southwark has shown that a) primary care is a very location based service and without fail each doctor exhibits a characteristic distance decay function that describes the pattern of registration with a GP suggest to some socio-economic criteria, but also that b) patients overlap to a large extent in a densely-populated urban context, the suggestion being that activity-spaces (i.e. retail areas, workplace and schools) has a distorting effect on patterns of registration for some people. To this end I suggest that a ‘community’ can be defined independently for individual GPs based upon the patterns of patient uptake unique to that service, although there may be some strong correlations with residential, workplace, educational etc. communities that overlap it (of course for some GPs the profile of its registered community may be greatly divergent from its observed local (defined by proximity to a GP) community). The following map is an example of this kind of complexity:

Here it is clear that any definition of locality or community based upon an arbitrary areal basis yields groups of people who could be registered to as many as 29 different Southwark GPs in only a very small area. This is in fact a very good, simple, illustration of patient choice in action. There are a lot of questions to ask Mr Lansley about how he views ‘local’ or ‘community’, and whether he is willing to enshrine that definiton in policy before we actually consent to doing anything with provision of services.

Further still, I have claimed that GPs are very much location based services – they are, over a certain distance (in Southwark this is about 6 -10km) no one is registered with a GP, choosing instead a closer service. In many ways this was constrained by the pre-existing system of ‘catchment areas’, however these were set to be removed by the end of the year in the quest for patient choice, thus the potential for registration is opened up to people using doctors near their place of work (for instance) rather than than near their home, thus should these people have a say in provison of services in the area within which they do not live – they are part of the GP’s ‘community’ but not of the residential one. A good illustration of this  is actually the polyclinic system – Southwark is geared up to introduce 3 polyclinics – one which already exists as a large GP-led health centre in the centr eof the borough, and two in the north connected to hospitals, the biggest difficulty faced at the moment is in estimating the daytime population (i.e. transient workforce) of the Southbank in order to account for likely polyclinic usage – a huge number of people who do not live in Southwark but will likely have some part of their healthcare provided for by Southwark PCT.

It is also unclear what Mr Lansley refers to when he talks about ‘top-down’: is it the Strategic health authorities and the DoH itself? It cannot be the PCTs as Mr Lansley claims that the new criteria will have the support of ‘GP commissioners’ and it is the PCTs that actually do the commissioning, further the idea of GPs working with local authorities is largely the same of GPs working with PCTs now, as PCTs and LAs are generally coterminous.

Whilst it is pleasing to see a politician quoting the need for an evidence based appraoch to restructuring, it is unclear what evidence he might base GP quality on, the current payment method (QoF) is based on GP reporting of pre-specified target outcomes to a centralised authority, surely GPs will simply follow these directives in order to bring in as much money as possible. Indeed, these stats are strong recommended not to be used as measures of GP quality as they are by-and-large patchy in what they cover, and include little demographic data. Indeed, had the previous government not already cut the NHS IT initiative that would have made reporting of outcomes actually feasible nationally, the new government would have no doubt cut it anyway.

The final worry I have is one of equity, something upon which the NHS is founded – the provision of a fair service contingent on those that need it, that is free at point of service. Surely such an atomistic approach to healthcare provision as Mr Lansley seems to specify, is liable to deepen the perceived ‘social gradient’ in health care, as without a careful (top-down) hand, the GPs and communities best-equiped to play an active role in orchestrating GP services will get increasingly better provision: most likely to be the wealthier areas of the country. There needs to be at least some form of national accountability for a national health service.

One of the best things about having address-geocoded an entire popualtion dataset it that you can finally get away from non-uniform areal representation (OAs, Postcodes) and present something that is uniformly disaggregate. Academics such as David Martin have long expounded the value of gridded representation of population data as it is regular and hence spatial unbiased. In fact the current work his group are doing is really interesting stuff, looking at daytime (as opposed to residential) population.

Anyhow, using the address-geocoded patient register for Southwark I was able to create a population density visualisation on a 100m x 100m grid that still preserves patient anonymity to an appropriate level. Of course there are some issues with Patient Registers, notably that they are only complete for people that register with a GP. Nonetheless, they provides a uniquely fine grain view of population in Southwark without resorting to the statistical uncertaintyof a smoothing surface-based density estimation, or an irregular, space-filling administrative/postal areal unit solution.

Yesterday Professor Mike Batty introduced me to the rank-size rule, an idea popularised by George Kingsley Zipf as the relationship between the frequency of an observed phenomenon against the phenomenon’s rank in its group. This is best exemplified by the example of city sizes: essentially Zipf shows that for every really large city, there exist many smaller ones; however these smaller cities aren’t just a bit smaller than the large city, they are considerably smaller, in fact the difference in city size from the biggest cities to the smallest can be explained by a power law, this can be represented as:

Where Pn is the frequency of occurance of a phenomenon ranked nth, and the exponent alpha is usually roughly equal to 1.

The power law thus produces a plot where the 2nd item is 1/2 the size of the 1st, the 3rd item is a 1/3 the size of the 1st etc. This can be represented by a plot of surname frequency in Southwark by rank.

Plot of Surname Frequency against Rank in Southwark for all observed surname (using R)

It is clear from the graph that there are very few surnames which are popular and many which are relatively unique. Another interesting characteristic of a power law, such as the relationship between surname frequency and rank are self similar: if we examine any portion of the curve we should get the same curve, albeit at a different scale.

Plot of Surname frequency for Rank 300 - 6000

It is clear from the above graph that a subset of the full data gives a power law relationship. We can attempt to linearise this relationship by taking the log of the frequency and rank:

The fact that the line is not straight indicates that the relationship is not a true power law. The long tail is accentuated by the stepped line, frequencies are integers so when we get to increasingly unique surnames the ranks tend to cluster. In the rank-size distribution of cities, the characteristic fall in the long tail when linearised like this indicates that city size distributions are really log-normal, however this is not the case in terms of surnames. If we exclude some of the long tail, the relationship can look a bit more linear as this plot demonstrates:

]]> http://danieljlewis.org/2010/03/05/some-surname-based-rank-size-thoughts/feed/ 0 http://danieljlewis.org/2010/03/03/analysis-of-surnames-from-southwark-patient-register/ http://danieljlewis.org/2010/03/03/analysis-of-surnames-from-southwark-patient-register/#comments Wed, 03 Mar 2010 15:32:51 +0000 Daniel Lewis http://danieljlewis.org/?p=243

My colleague James Cheshire’s research deals with understanding and classifying spatial patterns in surnames. He has been able to show, through various techniques, that there exists in the UK a regional geography of surnames. This in mind, I thought I’d interogate my database of NHS patient registrations for Southwark and see what was going on in surname terms there. This first table shows the top 20 most popular surnames in Southwark, ranked by occurance.

Figure 1: Top 20 Surnames in Southwark, by occurance.

Unsurprisingly perhaps, the top places are dominated by surnames native to the UK, classically Smith, Williams, Jones etc. However, in line with Southwark’s reputation as a diverse borough and in light of it’s high inmigration figures, it is also clear that of these top 20 surnames some of them would be connected to inmigrant names: Kamara, Ahmed, Ali, Patel and Khan are all surnames that are increasingly associated with a previous period of migration to the UK. Interestingly the Vietnamese population is very small, less than 1% of the population of Southwark, but around 23% of these have the surname ‘Nguyen’. The ethnicity of the surnames is derived from Onomap.

The frequency distribution of Southwark surnames looks like this:

Figure 2: Surname Frequency Distribution for Southwark, 2009

Note the characteristic long tail, there are a huge number of unique, or almost unique surnames, and considerably fewer surnames which are possessed by a large number of people. Such a distribution seems to obey a power law of some sort.

We can dig deeper into this phenomenon by looking at the number of surnames that comprise a given percentage of the population:

Figure 3: Surnames comprising given percentages of the Southwark Population

As we can see from the above figure, only 56 names account for 10% of the Southwark Population, but that in total there are 88,124 distinct surnames in Southwark. Again there is a characteristic decay to the curve.

Finally, let us consider just the charactersitics of the long-tail of the distribution:

Figure 4: Focus on the long-tail - percentage population for given surname frequencies.

From figure 4 it is clear that almost 25% of the Southwark population have a surname that is share by fewer that 11 people, indeed just over 16% of the Southwark population have a surname unique to the Southwark patient register. The shape of the curve in figure 4 demonstrate the effect of the long tail seen in figure 2.

For more information on surnames research check out James Cheshire’s blog, working paper or Pablo Mateos’ working paper.

]]> http://danieljlewis.org/2010/03/03/analysis-of-surnames-from-southwark-patient-register/feed/ 0 http://danieljlewis.org/2010/02/12/southwark-households-a-preliminary/ http://danieljlewis.org/2010/02/12/southwark-households-a-preliminary/#comments Fri, 12 Feb 2010 19:28:55 +0000 Daniel Lewis http://danieljlewis.org/?p=186

I’ve spent a chunk of time recently address geocoding the Southwark PCT patient register to Ordnance Survey Address Layer 2 data. What this means is that I can start identifying and (later) classifying households, this will allow me to ask questions about how different households approach healthcare. More broadly it allows me an insight into the demographic character of Southwark.

The data actually extends past the Southwark boundary as people in Lambeth, Lewisham, Bromley and Croyden do also to some extent use Southwark primary healthcare services (GPs) this means that although Southwark’s population is only c.300,000 the datset I’m using is for just over 340,000 people. There is some uncertainty in the data naturally, this results from the two datasets used; on the one hand addresses recorded in the Southwark patient register are not all necessarily complete, for example there is sometimes a failure to record which particular subdivision of a house someone lives in, or which flat in a larger block of social housing. On the other hand the AddressLayer2 data, although very rich, is not necessarily complete, this could be due to the prescence of unacknowledged subdivisions in residential housing, and although most social housing estates seem well documented, some commercial developments are not necessarily registered beyond the building level. Similarly, there are a number of instances of social institutions, such as the Salvation Army and St. Mungos, or marinas and dormitories having a single registered address for a high number of residents. This may have the effect of skewing the data slightly. With this in mind I created the following graph from the dataset of Number of households against number of inhabitants per household:

This shows that there is still a major trend for single-person households, but equally that around a quarter of all households are co-habited. The long tail in the graph (which i have truncated here) is caused by a few special cases, some examples of which are acknowledged in the previous paragraph. The average household size of 3.10 is itself higher than the UK average household sizes reported after the 2001 census which was 2.36; at the time the borough of Newham in East London had the highest household occupancy rate at 2.64. Of course there are any number of reasons why these data are not comparable, to start with the census took place 8 years before the Southwark dataset was created, similarly the uncertainty in the Southwark dataset is higher as it was not created with the primary purpose that it be able to successfully locate all patients as more often than not patients go to the Doctor and not vice-versa, whereas the census is distributed at a household level to each individual. The Southwark dataset does also include particularly tranisient communities which are missed by the census, such as the homeless who don’t have a fixed address (and hence may be using shelter or hostel addresses) but still require medical treatment at times.

Nevertheless, an interesting first look. The next steps will involve evaluating and validating the dataset to the best of my ability and then moving on to look at ways of examining and classifying household structure.