Python has been more tightly integrated in the new release of ArcGIS 10, allowing scripting to occur directly through a Python process without even opening up ArcMap. Admittedly this was available before, but now everything is more tightly coupled and a lot cleaner in it’s implementation. However, what has really interested, and indeed confused me of late is how to use Python in the ‘field calculator’.

Field Calculator is a really useful tool, when you are looking at an attribute table for a shapefile in ArcGIS and you want to derive a value for each object in the file based on a function you can input the function into the field calculator and it will work it out for you row by row. Sometimes the value you want to derive is a bit more complicated than simple arithmetic and you need to write a script. Previously you could do this in VBA, but I always found it limited and confusing, now however you can do it in Python – much simpler!

There are a few pitfalls to using Python in ArcGIS field calculator, and so I’m going to specify how to write simple field calculator python scripts in ArcGIS from my early experience.

Firstly, for Python in field calculator the way to do it seems to be in write a Python function, and then call it for each row. In addition to this, because you are writing a function you have to give it the relevant parameters (i.e fields) with which to do the computation. Finally, and annoyingly you have to write your function in a little box, and use a consistent indentation standard (1 space works best for reasons of space) as Python requires.

Here is a basic recipe for achieving field calculations in ArcGIS using Python. Obvious this is overly simplistic as you do not need a script to do this calculation, but it serves as an introductory example.

1) Name a function and parameterise it with the fields to base the calculation on. Do this in the lower box.

In the image you can see I’ve input: density( !sum_pop!, !Area!) This means send the values in the fields called sum_pop and Area to the function called density.

2) Define the function you are calling in the larger upper box.

You define a function in python using the “def” command. In the image i have defined the “density” function by writing the line: def density( pop,area):

This function definition means: define a function called density which takes the parameters pop and area. The parameters could be called anything, but it is useful to call them something that makes sense for use in the function. These parameters are variable names that the function uses to identify the fields you have passed the function when you called it, as in 1).

Normally you’d do some sort of calculation within the function, however this example is so simple that all we need to do is “return” a value to the function call. This function is the density defined as population over area: pop/area.

Looking at the field calculator I have found that you are limited to the basic, math and datetime modules in python, without the ability to import other modules. You can however define several functions and call them from within your main function.

For details on the basic syntax of using python, this site is particularly good: http://www.tutorialspoint.com/python/index.htm

We all know that representation in GIS is achieved primarily in either of two ways: with objects or with fields. This classic dichotomy was discussed at length during the 1990s when it suddenly became apparent that it is important to consider how the mapped representation of geographical phenomena actually changes our experiences of these in reality, or vice-versa.

If we look at a GI science text (the top of the pile being Longley et al, 2005) we learn quite simply that objects represent discrete entities that we might encounter such as cars, wheelbarrows, buildings, fences, roads or fruit. These discrete objects are best encoded as either a point, line or area (polygon). This gives the discrete object a kind of exacting presence in the GIS environment. On the other hand, things that aren’t discrete, i.e. they exist not as a single definable thing, but as a continuous ‘everywhere’, can be lumped into the ‘field’ category, which we know as ‘raster images’. These rasters are grids of cells which cover a defined area entirely and hold a value for each cell giving a defacto continuous field. Obviously using cells means it isn’t continuous, but lets ignore that and pretend we’re happy; in the end a good resolution will fool you into thinking that a raster represents a continuous space.

Now, vectors have generally risen to dominate the discourse of how we see the world. The cutting-edge products are all vectors, OS mastermap for example, and rasters have fallen by the way side a bit, as a social researcher it is easy to look at rasters as ‘something environmental GIS’ers do’. Nonetheless it is all too easy to find impassioned reasonings for the use of raster, or field, representations in GIS – as Helen Couclelis (1992) wryly states: “People manipulate objects (but cultivate fields)”. Thus there exists a tension in how we ought to think about representing our world and thus how we actually see it.

A Grid Cell with red flares representing the position of a subject when the neuron was 'on'.

Which brings me onto the main point: it seems that scientists in UCL’s Institute of Cognitive Neuroscience has established how the brain creates internal maps of new environments. The brain uses ‘Grid Cells’. The grid cells are actually special neurons which hold a sort of ‘spatial memory’, when you encounter a space these neurons fire in a sequence that, when mapped, show up as geometrically regular, triangular grids. Regarding this, Dr Caswell Barry said:

“It is as if grid cells provide a cognitive map of space. In fact, these cells are very much like the longitude and latitude lines we’re all familiar with on normal maps, but instead of using square grid lines it seems the brain uses triangles.”

So perhaps Helen Couclelis was right, we do use fields and luckily within GIS we have implemented any number of geometric rasters, obviously a square raster is standard, but triangles are not uncommon and neither are hexagonal matrices. What is more, the triangulated irregular network (TIN) acts as a kind of go-between, part vector object and part continuous field. Therefore, your brain on GIS is actually very closely related to your brain anyway! Perhaps a more field-based view of the world is called for.

References

The original article on this discovery is here.

Couclelis H (1992) “People manipulate objects (but cultivate fields): Beyond the raster-vector debate in GIS”, in Theories and methods of Spatio-Temporal Reasoning in Geographic Space, Lecture Notes in Computer Science, 639, Springer, Berlin.

Longley P, Goodchild M, Maguire D and Rhind D (2005) Geographic Information Systems and Science 2nd Edition, Wiley, London.

Michel Foucault introduces the idea of ‘heterotopias’ in his essay ‘of other spaces‘. In this work he first establishes that the present epoch will be the epoch of space: of simultaneity, juxtaposition, near and far, side by side and dispersed.

This pronouncement bodes well for geography as a discipline, but perhaps not for GIS. Foucault, as with other authors who could be thought to be ‘poststructuralist’, is against a determistic views of time and space; a view most evident in GIS where order in space is established through data structures and explicit object hierarchies. However, he also notes that the current understanding of places as ‘sites’ is linked to contemporary technologies.

Sites themselves don’t constitute space or place however, space and place are constructed by the relationships between sites. Thus GIS may have the capability to manage sites- which deal with the relationships and proximity between different objects as defined by structures (series, trees or grids) as well as the demographic issue of placing and attributing things within a site, i.e. siting- but not space and place itself.

Foucault defines heterotopias as sites which relate to all other sites, but suspect, neutralise and invert the relations that they have- basically a site which is linked ot all others but inherently contradict them. Foucault gives examples of heterotopias as sites such as hospitals, schools, cemetaries etc – sites in which societal norms are replaced by another norm which has the effect of challenging and inverting the societal norm and creates instead an ‘unreal space’. Another way of thinking of a heterotopia is as a reflection in a mirror- the image you see is a real place, but reflected to a virtual location and thus inverted.

From this, it is clear to see that we can think of GIS as heterotopias. Any given GIS, depending on the work that it is being used for, contains a set of sites that exist in the real world, but which, within the GIS, are reflections. These reflections are not real places, but abstractions and generalisations of real places projected in an absolute, virtual, space.

Foucault goes onto define some of the properties of heterotopias, most of which apply to GIS. Most heterotopias are heterotopias of deviation, this can be seen to be the case in GIS where norms of societies are replaced by mathematical and geometrical norms, similarly (as discussed in Pavlovskaya, 2006 and 2009 in a previous post) heterotopias are remade by societies as they change, as indeed are GIS. A heterotopia, like a GIS, is often linked to slices in time, taking everything up to a certain point, or simply taking a moment; it is this charactersitic in GIS that means that it has trouble handling temporal data. Heterotopias have rights of entry and are not freely accessible, perhaps this is also true of GIS which require a different way of conceptualising space to the norm. Perhaps the most important aspect though is the following: a heterotopia is a space which is ‘other’, it can be seen as the mirrored ‘perfect’ space- as well ordered as our real space is disordered. Thus the GIS cements its position as a tool for interpreting the conceptual model of our world, in doing this it is the ‘heterotopia of compensation’.

postscript: I’ve recently been reading Generation A by Douglas Coupland, the novel really dovetails together the ideas of unreality in things like maps and earth-observation images and the challenging of societal norms. The insight is similar and conguent with viewing GIS as a heterotopia.

I’ve been reading through a recent book entitled “Qualitative GIS: A Mixed Methods Approach” by Meghan Cope and Sarah Elwood, hopefully I’ll post a full review it soon. In the meantime however, I want to think about one of the sections in it by Marianna Pavlovskaya, specifically her discussion of whether or not GIS is inherently a quantitative tool. I tend to agree with Marianna on one front – rarely do most GIS users directly engage with the quantitative aspects of GIS, use of GIS becomes about spatial reasoning with overlays and logical tesselations of geometry and the impact of visual depiction. On the other hand though I offer a photo of books currently on my desk:

Figure 1: Some books on my shelf

I think there is a serious point here; GIS is a tool- people are happy to use tools to get the jobs they need doing done, I trust an allen key to undo parts of my bicycle without breaking them, and tighten them to a safe level. I don’t however need to know exactly how the tool is made, I just trust I know how to use it. The same is true of GIS, functions such as overlays are tools (in fact this is the exact terminology that ESRI use in ArcGIS), you use tools to analyse maps in a non quantitative way, because you reason and understand their usage; I don’t specifically need to know how the overlay works in a mechanical sense, just that it does. It is generally that case that such tools, even simple tools have a basis in quantitative fields, often mathematics; an overlay is a topological operator deriving from that branch of mathematics.

Now, in a GIS such tools are formed from solving challenges in ‘computational geometry’, and I agree with Pavlovskaya again in that ‘computerisation is not quantification’, except that in this case; that is exactly what it is. There are numerous instances in GIS, shown through point pattern analyses, topological functions and distance decay in which it is clear that GIS is a quantitative tool with a quantitative development.

So is Pavlovskaya wrong then? Well, no. There are different kinds of users in GIS and for the most part users aren’t looking ‘under the bonnet’, to them a GIS is a set of tools, the use of which causes them to construct (make) GIS as a non-quantitative thing. What seems strange is that in looking for opening for non-quantitative GIS in this perspective, the current way of constructing GIS quantitatively had to be challenged, or seen to be wrong and somehow misguided. In my eyes the interesting thing that comes out of this discussion is the idea of abstraction.

In a sense quantitative, or technical, ways of seeing are increasingly being abstracted: bundled and enclosed into something you can think of as a ‘black box’. This encourages engagement as the most common user experience is a qualitative one, as qualitiative experiences for most are more intuitive – generally we have the appropriate skills to deal with such experiences, but may need to learn the skills to engage quantitatively in a complex system such as GIS. Software such as GIS packages have the effect of making previosuly difficult quantitative functions much more accessible, even a measure of straightline distance is a quantitiative function – the Pythagorean theorum, and this can be done in Google Earth and Google Maps which are probably the most accessed GIS in the world (if not also the most accessible). I think the question we have to ask is: what are the tradeoffs in making quantitative functions more accessible, in having them reconstructed as qualitative tools? The obvious answer to this is that when things are conceived as ‘black boxes’ they are exactly that: we have no idea of what is going on inside them. Thus, qualitatively, we have to decide whether or not that is important; in fact for qualitative GIS is may not be that important, as the world is reseen with ‘fuzzy’ characteristics.

References

Cope, M. and Elwood, S. (2009) Qualitative GIS: A Mixed Methods Approach. Sage, London

(much of Pavlovskaya’s chapter also appears in the following paper)

Pavlovskaya, M. (2006) ‘Theorizing with GIS: A Tool for Critical Geographies?’ Environment and Planning A 38(11) 2003-20

Addendum

Muki Haklay often blogs about usability and critical GIS, amongst numerous other things of interest at his blog here.

I’ve been thinking about populations recently, perhaps unsurprisingly after the recent PopFest conference, nevertheless I tend to justify the presence of certain groups as characteristics of particular neighbourhoods or communities. In itself this seems entirely valid, certainly no population is perfectly homogeneous, they exist along a spectrum of heterogeneity. We can see simply from our own experience of the world that some areas exhibit spectacular diversity of inhabitants, particularly urban areas, my case study area of Southwark, London springs to mind. Situated in central London, south of the River Thames it contains a population that spans the inequality divide from the wealthy, young, City workers resident along the redeveloping ‘Bankside’ and the older, settled familied of the leafier southern part of Southwark, to the poorer, often non-white, communties in central Southwark living in social housing, to the poorer, older white residents historically attached to the docklands to the north-east of Southwark. Notably rural areas tend toward greater homogeneity, but still exhibit some diversity in some characteristics even if others, particularly ethnicity are constant.

This is, however, not the point, the real issue is the difference between neighbourhoods and communities. Both are very difficult to define clearly, my snap judgement is that communities realte to people and neighbourhoods to the territories that people inhabit. However some deeper research will show cracks in this simplistic approach, sure neighbourhood usually relates to a metric space, things that are near to, or contiguous to something else, however creating neighbourhoods for individuals seems like an approach to constructing communities for those individuals, likewise geodemographic classifications are increasingly being conflated with indicating something signifcant about the underlying communities. Similarly, communities occupy particular areas, the practice of mapping communities which come from government requirements seeks to territorialise community and new pratcices in public service causes further confusion: community policing for safer neighbourhoods?

The main rule that I have been able to extract from much of this is the propensity for communties to represent a ‘softer’, perhaps more public facing approach and conversely, by neighbourhoods a ‘harder’, more quantified method is often implied. There are notable splits within the disipline of geography between those who are willing to engage ‘communities’, which are intangible, temporally variable, socially realised relationships (perhaps) and come with the baggage of uncertainty and those that would rather deal with the geographically discrete entity of the neighbourhood, safe in the knowledge that the implied precision of areal units are not widely questioned. Even within GIS there has been a diversification of approach, from the mainstream GISci enforcing a ‘spatial science’-style approach that has persisted since the quantitative revolution (at least), to the public participation GIS discourse which actively engages communities and individuals, and by extension critical GIS which for a time has critiqued territorially bounded ‘descriptions’ of community. Finally consider the governmental binary of ‘neighbourhood statistics’ and ‘community cohesion’, many councils use neighbourhood statistics to ‘measure’ community cohesion!

Recently I’ve been doing some work involving the transportation problem. The Transportation Problem is an allocation optimisation problem that requires the optimal assignment of demand, in my case patients by Output Area, to known, fixed, supply points, in my case doctors surgeries (General Practices). Rather than use a euclidian or manhattan metric to model distance from the demand site to the supply site I have used public transport travel times from TfL.

Initially this seemed a difficult task, and early attempts only provided partial, or non-optimal, solutions. However, once I had found the linear programming functionality in R through the package LPSolve it became very easy to create a model with the constraints I wanted and get a solution very quickly. Key to the success of the R package was the ability to set the constraints I needed, crucially integer constraints so that people were not subdivided, and constraints on the number of patients doctors could take.

Mapping the outcomes in ArcGIS was straightforward due to R’s built in csv-export funtionality.

Here is an example of the output.

The legend denotes the 44 physical practices in the London borough of Southwark, some doctors exist on the same site and so these practices were agglomerated. The grey areas represent unallocated demand caused by capping the size of the General Practices. In another definition of the model I ran I set the GPs up as uncapacitated so that all the demand would be satisfied. This model uses data from 2006, I have data from 2009 for which I will also run the model.

Some earlier work I did on this is available in the Proceedings of GISRUK ’09.

The above is a quotation from the book “Foucault’s Pendulum” by renowned author Umberto Eco, in the context of the book the phrase relates to a discussion that is had regarding homunculi (singular: homunculus), literally ‘little humans’, which are being kept in a garden in vessels of liquid. When conversation errs towards a human treatment of these creatures, one member of the party remarks “they are only cartesian imps. Or automata”, essentially, it is not appropriate to feel responsibility for these creatures because they are not humans, merely representations of humans.

This led me to consider the position of Geographic Information Science, the ideas of cartesian imps (agent based models?) and automata (cellular automata?) seem to resonate some of the interpretation of Eco’s text, and it highlights some interesting debates. GISci relies on representations and models of reality, it cannot represent the true thing, it was made to generalise, and hence perhaps is a homunculus, the victim of a diminished status in the eyes of those seen to represent the true thing, the real world, in all its relational complexity.

In the Eco text the brief interlude of the homunculus is quickly forgotten with a return to a more established literary tradition: desire. However, it is less easy to turn our back on the same issue in GISci, can we conceivably endorse such a view, that GIS is just a system, a machine for geography? This is the perspective that led to the fiery debates in the 90s between human geographers or social theorists and the GIS community, the tool of GIS was divorced from a human nature, it had no ethic, no morality. This perspective is gradually changing, there is greater engagement with mapping through the naive geographies offered by google maps and co, likewise the GIS establishment have begun to embrace a humanising agenda, from VGI to reconceiving of GIS as media- communication is a powerfully humanising force- to the growth of a critical community that is seeking to rebuild and adapt GIS to a more social means from the inside, not from without. Increasingly GIS is moving away from its positivist roots to include a greater appreciation for human methodologies, one such being the ability of the public to challenge their geographic classification and feedback their experience into the system. Sure, people are still researching voronoi polygons and data structures, but increasingly it seems these projects are being realised in a form that in publically-readable, not merely expert-readable, or (god-forbid) only machine-readable.

I notice that there are stirrings within the GIS community of “the quantitative revolution 2″ (see Mei-Po Kwan and Tim Schwanen), which whilst sounding like a tacky sequel may herald something more than a large budget and meaningless special effects. The tag line of this revolution is: “The critical (re)turn”, I think this is essentially what is required in GIS, and it has been happening anyway, to me the ‘critical (re)turn’ means the inclusion of mixed-methodologies, engaging with people and having a more open and involved discourse. Of course we shall have to wait for Mei-Po and Tim’s paper in a forthcoming Professional Geographer to see whether this is the case, or whether there is something fundamentally more important to say.

From all of this, I guess the question is: to what extent have we humanised the homunculus? or, will we ever get over the fact that it is a homuncuus rather than a human?