Processing UK Environment Agency LIDAR Data Tutorial

Recently, the Environment Agency has released its LIDAR data to the public. This is not the first time that DEM (Digital Elevation Map) data has been released from free. Satellite based DEM data has been released by ASTER and SRTM (SRTM is less noisy), but those have a horizontal resolution of 30m, which is quite coarse. The EA data, which was collected using LIDAR, has a much higher horizontal resolution, but the downside is that there is gaps in the data. Several of my archaeology friends have asked me to do tutorial on how to display this data.

If you look at the website, zoom in and click on a map tile, you can see that you can download 6 different sets of data. There is DTM (Digital Terrain Map) and DSM (Digital Surface Map) data. The difference is that the DTM data has stuff like trees stripped, which makes it much more useful for archaeology than the DSM data. It also comes in 3 resolutions, 0.5m, 1m & 2m. The trade off is that there is less area convered for the higher resolutions. I would recommend starting with the 2m data and downloading 1m for the same area if you really need it.

When you download a file, you will get a zip file containing up to 100 ASCII grid files, each covering a 1km square. This is not a normal image, but a file containing the height data. which we must process using GIS software. This tutorial will cover processing that data using a popular open source GIS package called QGIS, which you can download for free as a standalone application or as part of the OSGeo4w package. Here are the steps to follow.

If you haven't already got a project in QGIS, create a new one, setting the map projection to OSGB36 (EPSG:7405).

Unzip the contents of each zip file, and place the contents of each one in a folder of their own, for example, if your zip file is called LIDAR-DTM-2M-TR14.zip, create a directory called LIDAR-DTM-2M-TR14 and copy the contents of the zip file in there.

You will notice there are a lot of files, and we don't really want to deal with them all individually, so we will merge them together into one giant ASCII grid, covering a 10x10km square. To do this in QGIS, go to the Raster > Miscellaneous > Merge... menu. Next to the Input Files, press the Select... button and select all of the asc files in a directory. You also need an Output File for the merged data, for which I use the name of the directory again, which must be followed by a file extension of .asc, for example LIDAR-DTM-2M-TR14/LIDAR-DTM-2M-TR14.asc, preceeded of course by the rest of the file path to that directory. Finally, tick No data value and set it to -9999. Press ok and it will start working, which may take a while. It should load into QGIS automatically, having asked you what projection to use (OSGB36 again), but sometimes the process crashed. Don't worry, because the creation of the file has completed, so back on the QGIS main screen, press the Add Raster button and load the newly created file. Having merged all the files, you can delete all the original small unmerged files if you want, since they are pretty big. By default, you get the data displayed as a grey scale image with white as high and dark as low, as shown below.

You will notice that if you have more than one of these next to eachother, the edges don't match, so we need to set them to have a matching palette. We will also use a multi colour palette to provide a bit more contrast. Pick an image, right click it in the project window and select Properties. Go to the Style tab and change the Render type to Singleband pseudocolor. Then use the + sign to add bands. I currently use 0m-Black, 20m-Green, 40m-Brown, 60m-Orange, 80m-yellow and 100m-white. You can of course make up your own palette to suit your tastes. It should look something like this.

You can then right click on your image in the project window and select Styles > Copy Style and then right click on the rest of the images to do Styles > Paste Style to avoid entering that palette information all over again. This should give you something like this.

This shows the height information very well, but if you want to see small changes that might be created by archaeology, then we need to add a hillshade to this. To do this, go to Raster > Analysis > DEM. In the Input file, select one of your merged files. For the output file, use the same filename, except add -HS just before the .asc. You will notice that the resulting file will completely obscure the height data below. To fix this, right click on your newly created hillshade image, go to Properties, go to the Transparency tab and move the slider until it is at 50%. You should now be left with something like this. If you zoom in with QGIS, you will see a lot of the smaller features. Happy hunting!

Green Waste, A Growing Problem

A relatively new, but increasing, bane for geophysicists and metal detectorists alike, is the practice of spreading green waste on fields. Part of the drive for increased recycling in society, which is good, this material comes from our gardens, but is rarely pure. Many households are not too fussed with what they will put in their bins, so plastic and metal will end up in the green waste. Farmers will buy this from the council and then use it to fertilise their fields. The metal component of this will cause problems for magnetometers, producing white noise from thousands of tiny dipoles. For example, this is a Roman settlement :

This is a Roman villa :

Once it's in there, it's there for good. Metal detectorists get it worse, as they are also picking up non-ferrous material. There is even a blog dedicated to the problem.

Latest Results: Chichester

This year, my biggest project was spending a week within the Chichester town walls, looking for Romans with the GPR. The original purpose of the visit was to test the theory that Stane Street did not start at the east gate, but actually went through the Roman town and out the other side. It soon became clear that the archaeology was too shallow to be visible amongst the modern services. Instead, I ended up surveying some grassed areas to look for signs of the Roman town. This is just a quick summary, but you can read the full report here.

There were two main areas I surveyed. The first was in Priory Park, in the NE corner of the Roman town. The survey revealed two Roman buildings south of the Guildhall and part of the Roman road grid internal to the town to the east, with some low status settlement. The road had been cut in two places by the medieval motte and bailey ditches. You can see a video of the results here and of the two buildings here and here.

Priory Park Interpretation. Click for larger image.

The second area surveyed was the amphitheatre, just to the south-east of the Roman east gate. It isn't in particularly good condition, most of the retaining walls have been robbed, but some structure is still visible. You can see a video of the results here, but on that version, a block of data is shifted from where it should be.

Amphitheatre Interpretation. Click for larger image.

While these are excellent results, they are nothing compared to what is currently being found at Verulamium, where they are using mag and radar to reveal the structure of the town. I'd like to give a shout out to their most excellent blog, which shows the results in great detail. Well worth a read.

Digging Up The Geophysics: 2015

The new archaeology digging season is underway, but is far from over! If you fancy having a dig this summer, here are some sites in Sussex under excavation on which I have previously done geophysics. At the end of the season, I will do another post about what they have discovered.

Ovingdean

Brighton and Hove Archaeological Society are again digging at the medieval site at Ovingdean, where are are uncovering part of the enclosure around the manorial enlosure. What they have looks much more substantial than a simple retaining wall. Are there more buildings against the edge of the enclosure?

 

 

Plumpton

The Sussex School of Archaeology are running a training dig at Plumpton Roman Villa. The digs run during the week and are targetting the eastern part of the villa.

Barcombe

Culver Archaeology Project are digging again at Bridge Farm, where they are targetting the intersection ot two Roman roads and the defensive enclosure around the centre of the settlement. When I visited, they were exposing a possible cremation burial and had found an intaglio. Their site has a weekly blog to keep up with what is going on.

Version 1.15 of Snuffler released

A small update to Snuffler this time around, though bigger things are already in the pipeline for the future. This time sees the addition of an import for files generated by the new Frobisher TAR-3 earth resistance meter. Low cost options such as this are the way that community archaeology groups get started with geophysics, so it is always nice to see new options on that front. I haven't used the new hardware myself, so I can't comment further on it, but I wish them all the best in their new endeavour.

You can download the new version at the usual place.

This blog has been fairly quiet as I have been working on some big new projects and finishing work already reported on here, which you will hear about later. In the mean time, here is a picture from the Severn Sisters, where the National Trust are running an archaeological project this year. This is from Bailey's Hill, where some possible Bronze Age remains are due to be excavated in August. If you would like to get involved, you can!

New Toy: Survey Grade GPS

I have a new toy! Oh joy, a new toy! This isn't actually a new piece of geophysics equipment, but survey equipment. Having previously set out grids using a total station and recorded data on an arbitrary grid, I can now get absolute coordinates, set grids out much quicker and with only a single person. I went for net rover rather than rover plus base as I wanted decent coordinates in the field rather than post processed.

I eventually chose a Javad Triumph-LS. There were several things that attracted me to this particular model. It received all signals from all constellations (so futureproof and accurate), it was cheap (relatively!) and the company had made an effort to deal with interference due to the lightsquared debacle, which is good for me, because part of its job will be sitting on top of my radar antenna. The unit even gives you a relative quantification of interference it is receiving. I put it on top of my radar, turned off, and the value was 4. Turned on, it was 22. Rather than having the receiver on top and data collection half way down the pole, both are combined at the top of a mono-pole. It does seem a bit top heavy and difficult to keep steady, but it will correct itself for pole tilt, which is a nice feature. Control is via a number of hardware buttons and a capacitive touch screen like a smartphone, so the whole thing seems very modern. Getting started with the kit, the support on their forums was excellent. On the downside, I found a few things unintuitive, but most of those can be put down to me being a geophysicist rather than a land surveyor. The receiver has not long been released and the manual still needs a bit of work to deal with the likes of me.

So that's the hardware, which I'm very pleased with. I also needed a network RTK corrections subscription. In the UK, all of the correction services are based off of the base stations run by OSNet from the Ordnance Survey. Of the five different services that use OSNet hardware, two are aimed at farming. While these are cheaper, they only connect you to the single base station, which is fine if you are near one, but not very good if you are not. You can only expect to get 5cm accuracy with these services. The other three services, SmartNet from Leica GeoSystems, VRS Now from Trimble and TopNET live from Topcon do things differently. They create a model of the atmosphere based on a number of base stations and create a Virtual Reference Station based on your current location, giving you a potential accuracy of 1cm. Apparently all three services produce a similar level of accuracy, but they don't give any information on which constellations/signals are corrected or whether they provide any additional base stations over and above those provided by OSNet. Price wise, at the time of beginning to look at all this, their websites showed the Leica and Topcon services charged £1200 ex vat per anum for the limited (40 hour per month) service, while Trimble charged £1500 or £1300 sans SIM card.

Trying to sign up to one of these services turned out to be a tale of woe. I started out contacting Leica. The contact email address and price on the website was wrong (now fixed) and the price is now £1260. After speaking to someone by phone and then emailing, contact went dead and they stopped replying to my emails. Not very good if they don't even want to sell you something. I next tried Topcon. I filled in their web form and got an email saying that my registration was confirmed... then heard nothing for a week. Next I tried Trimble. I filled in their web form, which then demanded that I give them my VAT number. Not even having a company, I don't have one of those, so Trimble fell at the first hurdle. They were too expensive anyway. I went back to Topcon, having noticed another contact email in their confirmation email. Once I had the attention of a human, service was prompt and helpful. There were further problems, like the first SIM card they sent got lost in the post and the second one was the wrong size (mini instead of micro), so I cut that down to size, but I messed it up and cut it slightly wrong.  They sent me a SIM of the correct size and all was well in the end, so Topcon came out on top(con). Connecting to the service, I got my desired 1cm accuracy. According to the receiver, it was receiving correction for GPS + GLONASS. Hopefully, the Ordnance Survey will upgrade their base stations to cover Galileo now that constellation is coming online.

One thing I wanted to test is that the receiver supported the black abomination that is OSTN02, a modification of OSGB36. I surveyed a random point in ETRS89, converted it internally to OSGB36 and did the same conversion with the Ordnance Survey's own coordinate transformation tool. Initially, it seemed it didn't, as the difference was about 2.06m. It turned out that I also had to change the datum used by OSGB36 to the Newlyn datum, which made it produce the same values as the OS site, so all good there. I also wanted to test how accurate Google Earth imagery was locally, so I recorded a bunch of points in ETRS89, converted them to WGS84, as used by Google Earth and converted it to kml. Here is a picture of how that turned out. Looks like the Google Earth imagery is very good indeed!

How do I use this fantastic new toy in the field? With the total station, I used to set up the total station at one end of a baseline along a straight edge of a field, tell it that it was at 500E,500N, point it along the baseline, then tell it that it was facing something like an angle of 270 (west), even though it wasn't. Then I could just go and look for 460E,500N etc. For reestablishing the grid, I used to record two resection points that could be described to a few cm, the downside being that those points could disappear, which did happen in a couple of cases. For placing on maps, I recorded a bunch of points at the edge of the field to match to the edge of the field on maps or aerials. The process is described in more detail here.

With the GPS, I start by collecting the same point I originally would have occupied with the base station, then setting it to stake out to that point and walking 120m away along the baseline. I then do a Multiple Point Localization on those two point, using coordinates of 500E/500N and 380E,500N, which will create a new projection which I can use to walk to further points just as with the total station, only with a single person instead of two. The two points used for the localization are recorded in lieu of resection points, but the new projection is stored on the device, so they are only needed if other people are re-establishing the grid. Since I get absolute coordinates, there is no need to record points around the edge of the field to overlay the geophysics correctly, so everything becomes much faster and more reliable.

Now to get out and use it. I'm going to be doing a big radar survey of Chichester in July, so watch this space for some results from that.