More

How to run GAM with raster package?

How to run GAM with raster package?


When I am using the raster package to predict a surface probability map of species distribution using the following functions in R, the following error occurrs:

CA132122200=raster("F:/CBG maps-UnProj/anlytic final map/mov472/landmergbackg1.tif_mw2/ca_132.tif") mod=gam(y~s(CA132122200) ,data=training, family=binomial, gamma=1.4, logLink="logit") prediction <- predict(object = CA132122200, model = mod, type="response", progress="text") errors: object is not a matrix: In addition: Warning messages: 1: In predict.gam(model, blockvals,… ) : not all required variables have been supplied in newdata! 2: 'newdata' had 2032869 rows but variables found have 2315 rows"

How to use raster package in python with rpy2

Working with R in Python using rpy2 on windows 7. I need to open some rasters as RasterLayer using the function raster() from the raster package. I manage to install the package, but not to use its function.

I install the packages that I need (rgdal, sp, raster, lidR, io) using

names_to_install is a list of the packages that are still not installed. This works fine.

I know how to try the "basic" functions, like sum, and it works:

But the same doesn't seem to work with the raster function from the raster package:

I also tried the following:

with the intention to be able to run the next and load my raster file:

but the first line (import('raster')) causes the crash of python and I get the error:

This doesn't happen with other loaded packages like rgdal, but with the raster package and with the lidR package I get the error.

I looked up this error, seems to be access violation, but I don't know what I can do about it and why it only happens with certain packages.

I expect to be able to call the raster function from the package raster.

I tried it on a computer with windows 10 and the error doesn't show anymore when running

Still would be nice to know what is the problem with Windows 7 and if there is any solution.


How to run GAM with raster package? - Geographic Information Systems

This package implements various tests, visualizations, and metrics for use with environmental niche models (ENMs) and species distribution models (SDMs).

Warren, D.L., Matzke, N.J., Cardillo, M., Baumgartner, J.B., Beaumont, L.J., Turelli, M., Glor, R.E., Huron, N.A., Simões, M., Iglesias, T.L. and Piquet, J.C., 2021. ENMTools 1.0: an R package for comparative ecological biogeography. Ecography, 44(4), pp.504-511.

ENMTools is available on CRAN, and can be installed by simply typing:

You can also install the newest version from https://github.com/danlwarren/ENMTools. There are multiple ways to download it. The easiest is to use devtools and install from GitHub.

Installing from GitHub using devtools

Run the following code from your R console:

If you want the absolute newest version you’ll want to install from the “develop” branch, but be aware that sometimes things get broken on there and it may take us a little while to notice.

A zipped version of the package is available at https://github.com/danlwarren/ENMTools/archive/master.zip. To install from the zip file, download a copy of it to your system. Once it’s finished downloading, type the following (where PATH is the path to the zip file):

ENMTools uses functionality from a LOT of other R packages, and it’s possible that you don’t want to install them all. For that reason many of the packages are not automatically installed with ENMTools, but instead “suggested”. If you want to install all of the suggested packages, we have a function for that. You should only need to use it after you first install ENMTools or update R. If you choose not to install the extra packages, you will get warnings when you try to use functions that require them.

Interacting with ENMTools

Creating enmtools.species objects

We’re going to load in some environmental data. You can do this from local rasters, like so:

Or you can load them from the internet using the raster package’s getData() function.

Regardless of where you get your environmental data from, it’s a good idea to call the check.env() function on it. This makes sure that an NA in an environmental layer propagates to the other layers, which can avoid issues with some of the modeling functions.

ENMTools is primarily designed to examine patterns of similarity and difference between ENMs for different species. In order to simplify interactions with the functions in ENMTools, you need to put your data for each of your species into an enmtools.species object. You can create and view an empty enmtools.species object just by typing:

You can add bits of it when the object is created:

Or you can add data to this object after it’s created:

It’s always a very good idea to run the check.species() function on an enmtools.species object after you build or modify it.

And now we can take a look at our species!

However, ENMTools also contains some sample data. It contains an enmtools.clade object called “iberolacerta.clade”, which holds several enmtools.species objects including an already-built version of Iberolacerta monticola. It also has some low-resolution Worldclim data that we can use to demonstrate functions. We’ll pull two of those species out now.

ENMTools contains functions to simplify the ENM construction process. Using enmtools.species objects and the correct modeling commands, we can build models very quickly. These commands are primarily wrappers to dismo model construction and projection functions, and at present are only available for GLM, Maxent, Domain, and Bioclim models. One of the nice bits about this setup is that it allows enmtools to automatically generate suitability maps, do model evaluation, and plot the marginal suitability of habitat for each variable separately.

Before we get started modeling, however, we might want to trim our predictor set so as to reduce collinearity. ENMTools contains a couple of handy functions for this.

That’s great, but it’s a bit hard to pick variables this way. Let’s try it visually instead.

The raster.cor.plot function gives us two visualizations. One heatmap that colors pairs of predictors by their Pearson correlation coefficient, and one cluster plot that does mds scaling of the predictor variables and then plots them in a two dimensional space so that more correlated predictors are closer to each other. We’re going to make an arbitrary decision to just use three predictors, and to keep those predictors relatively uncorrelated we’ll select predictors that are far apart in this mds plot. Here we’ll choose bio1, bio12, and bio7.

GLMs usually require the user to supply a formula, an enmtools.species object, and some environmental data. If your formula is a strictly additive function of all of the environmental layers in env, though, enmtools.glm will build a formula automatically.

Notice this produces the same formula as:

If you want a more complicated formula, though (e.g., with interactions or polynomial effects), you’ll need to supply that manually.

To check out the marginal response functions, you only need to type

These plots present a smoothed estimate of the frequency of different levels of the environemntal variable in the presence data and the background points, along with the estimated relationship between that environmental predictor and the suitability of habitat from the model.

You can also visualize your models and data in a 2D environment space using any pair of layers from your environment stack. These plots hold all non-plotted variables (bio7 in this case) constant at their mean value across all presence points, then vary the plotted variables between the minimum and maximum values in env.

The suit.plot shows you suitability in environment space as a function of your two variables, with brighter colors representing variable combinations predicted to be more suitable. The points represent the occurrence points for your species in that environment space.

The colored raster of the background.plot shows you the density of background points in environment space, while the white points again represent your occurrence points in environment space.

GAM, Bioclim, Domain, and Maxent

The procedure for building Bioclim, Domain, and Maxent models is similar to the procedure for GLMs, with the exception that you do not need to pass a formula to the model function for Maxent, Domain, and Bioclim models. Note that running Maxent models requires a bit of extra setup see dismo documentation for details.

Metrics: breadth, correlation, and overlap

ENMTools provides a number of metrics for ENMs and for similarities between ENMs. These include measures of niche breadth, based on Levins(1968). An important caveat when interpreting these metrics is that they are driven to some (variable) extent by the availability of different combinations of environmental predictors. As such they are more accurately interpreted as a measurment of the smoothness of the geographic distribution of suitability scores than as an estimate of the breadth of the fundamental niche an organism with narrow fundamental niche breadth that nonetheless encompasses a set of environmental conditions that is quite common will have a high breadth when measured using ENMs, while having a low breadth in environment space.

ENMTools also provides metrics for measuring similarity between ENMs. These include Schoener’s D (Schoener 1968), I (Warren et al. 2008), and the Spearman rank correlation coefficient between two rasters. While D and I are commonly used in the ENM literature, they may tend to overestimate similarity between ENMs when many grid cells are of similar values (e.g., when two species prefer different habitat but the region contains a great deal of habitat that is unsuitable for both).

A new feature of the R version of ENMTools is that you can now use these same metrics in the n-dimensional space of all combinations of environmental variables, instead of restricting your measures of model similarity to those sets of conditions that appear in the training region. This is done by repeatedly drawing Latin hypercube samples from the space of all possible combinations of environmental variables given the min and max of each variable within the training region. ENMTools continues to draw samples until subsequent iterations differ by less than a specified tolerance value. Lower tolerance values result in more precise estimates of overlap, but can take much longer to calculate.

The plots that come out of these environment space functions are used for diagnosing convergence of the overlap/breadth metric. Ideally what you want is a relationship between the metric and the number of samples that shows no clear directional trend.

Niche identity or equivalency test

In this example, we will run a niche identity (also called equivalency) test, as in Warren et al. 2008. This test takes the presence points for a pair of species and randomly reassigns them to each species, then builds ENMs for these randomized occurrences. By doing this many times, we can estimate the probability distribution for ENM overlap between species under the null hypothesis that the two species’ occurrences in the environment are effectively a random draw from the same underlying distribution. Note that niche evolution is only one of many reasons why two species’ realized environmental distributions might cause departures from this null hypothesis. See Warren et al. 2014 for details.

To run an identity test, we need to decide what type of models we will build, how many replicates we will run, and (in the case of GLM and GAM) a model formula to use for empirical models and the Monte Carlo replicates. The resulting object contains the replicate models, p values, and plots of the results. Typically idenity tests are run with at least 99 replicates, but we are using a smaller number here for the sake of execution time.

NOTE: In order for models to be comparable, both empirical and pseudoreplicate models for the identity test are conducted with pseudoabsence points pooled for the two species being compared.

Background or similarity test

The background or similarity test compares the overlap seen between two species’ ENMs to the overlap expected by chance if one or both species was effectively choosing habitat at random from within their broad geographic range. The purpose of this test is to correct for the availability of habitat and ask whether the observed similarity between species or populations is significantly more (or less) than expected given the available set of environments in the regions in which they occur.

NOTE: In order for models to be comparable, both empirical and pseudoreplicate models for the background test are conducted with pseudoabsence points pooled for the two species being compared.

In Warren et al. 2008, we developed this test in the context of comparing one species’ actual occurrence to the random background occurrences of the other species. This is what we call an “asymmetric” test, and in our case we did the test in both directions with the idea that we might compare the results of A vs. B background to the results of B vs. A background. This may be informative in some cases, but many people have also found this asymmetry confusing (and indeed it is often difficult to interpret). For that reason, the background test here can be conducted against a null hypothesis that is generated from “asymmetric” (species.1 vs species.2 background) or “symmetric” (species.1 background vs. species.2 background) comparisons.

Here, for instance, is a Bioclim background test using the classical asymmetric approach:


How to run GAM with raster package? - Geographic Information Systems

Quantum GIS Raster Array Module

A Python module for manipulating geographic raster image data in QGIS using numpy arrays. Includes a demonstration for Conway's Game of Life.

Dependencies: QGIS 2.6.1 Brighton, Python 2.7

QGIS Python Console Setup

Go to the python command console in QGIS
(Plugins > Python Console)

Enter in the console:
import sys

Find the location path of this script
C:UserDesktop

Append it to the QGIS system paths with:
sys.path.append(FILE_PATH_HERE)
sys.path.append ("C:UserDesktop")

Import the Cells class for use in the console:
from RasterArray import *

A. Creating the Cells Object

Create a Cells object as a random array:

Create a Cells object as a filled array:

Create a Cells object as a custom array:

  • r1c1..rncn refers to row 1 (r1) and column 1(c1) to row n (rn) and column n (cn)
  • The containers [] define a list and () define a tuple

Create a Cells object with a raster file:

cellsObject = Cells ("path_to_raster_file")

B. Modifying and Obtaining Cell Values

cellsObject.modify(x, y, value)

  • x and y are the coordinates of the cell to modify
  • value is the value to replace the cell value at the location
  • for non-geographic coordinates, use (x, y, value, geographic=False)
  • x and y are coordinates of the cell to obtain
  • for non-geographic coordinates, use (x, y, value, geographic=False)

Output a raster representing the changes:

Set the band of the raster

cellsObject = Cells (nband=n)

Set the spatial reference system of the randomly generated raster

cellsObject = Cells (EPSG=coorSys)

Setting the dimensions and cell measurements of the randomly generated raster


Updated 2017-02-24:

I think that the best solutions is to use R with RStudio. ( Python and iPython notebook is an alternative ):

  • Data import
    • Excel: readxl package
    • Oracle: ora or RODBC package
    • Copy-and-paste: RStudio's exports plot functionality
    • Programatically: ReporteRs package

    Data Import

    There are numerous ways to import excel (tabular) data. For Excel data, the readxl package provides the easiest and most versatile. It generally gets the variable types correct on import.

    Alternatives are to save the file as CSV and re-import. The readr package is good for this. @Nick Stauner provides perhaps the most basic solution using read.csv the limitation is that this requires the additional step of saving a worksheet as a CSV file. This is not great if your data is spread across multiple sheets. It can get tedious though there are VBA programs for saving all sheets as CSV files. Google for them. Another limitation is getting the types of the variables correct. If you use read.csv , you often have to fix your types after importing in R.

    There are a few packages that avoid these problems by allowing you to connect read/write from the spreadsheet directly or by using ODBC. Search on CRAN for excel or odbc to find the relevant one for your situation.

    Plotting

    In terms of getting plots into powerpoint, use Rstudio's export plot functions, the copy and paste method using Rstudio is:

    export plot > copy plot to clipboard > copy as: metafile captures the plot to the the paste buffer allowing you to paste directly into Power Point.

    As far as generating plots, R has numerous options. The aforementioned ggplot2 package provides a very powerful interface for creating all sorts of plots. There are additional packages for doing hundreds or thousands of other types of plots/animations/etc. One limitation is that these are often buried in CRAN packages.


    Use ArcGIS Pro to make an offline map - part 1

    by MarkBockenhauer

    Fieldwork takes you and your mobile workers places without a data connection, but that shouldn’t stop you from using your GIS data. You can make a map that doesn’t require the internet, and you can use it on your device in the field. You might be using Explorer for ArcGIS, or perhaps a custom application built with AppStudio for ArcGIS or an ArcGIS Runtime SDK. In this blog, we’ll look how to build a map that can be used offline (without a data connection) in any of those apps. We’ll then look at using that map in Explorer.

    First, let’s see an example of an offline map using Explorer for ArcGIS . If you don’t have the app installed already, start by installing it on your iOS or Android device. Search for Explorer for ArcGIS in the App or Play store.

    Start Explorer and tap Continue Without Signing In when it launches.

    Then search for “Shenandoah National Park” and tap to download the map.

    Check out the map to get an idea what is possible: tap features and see information about them, view the map’s layers, measure, or draw markup on the map. If you are new to Explorer, take a look at Get started and try some of the things called out there using this offline map.

    Now we’ll use ArcGIS Pro 2.1 and build an offline map like the Shenandoah National Park map. Behind the scenes, an offline map is a Mobile Map Package (MMPK). As you create the MMPK by following this blog, you’ll learn the recommended pattern for creating an offline map (or MMPK) like the one you just tried in Explorer.

    In this example, we are going to start with a basically finished map. To do so, we’ll get an ArcGIS Pro project that already has some completed maps in it.

    Note: This exercise requires ArcGIS Pro 2.1.

    Estimated time to complete - 40 minutes.

    1. Start ArcGIS Pro and click Open another Project
    2. Click Portal and then Browse to get a shared project from ArcGIS Online.Click the All Portal filter and search for “Make Offline Map example.” Select it and click OK .

    3. The project downloads and opens with three maps (often called views) – Operational, VTPK_Source_BM and Basemap. The Operational map consists of places, points, and areas of interest. The layers in this map provide searchable and clickable features for your offline map. You can author the Operational map to allow your map users to search for features by attributes and to tap on features to see pop-ups.

    Click on the tab for VTPK_Source_BM to activate the basemap view. It shows the boundaries for Yellowstone and Grand Teton National Parks. The information in this map provides background reference data for the operational layers. Layers in the basemap are purely for visualization. Users of the offline map won’t be able to access attributes through a pop-up or be able to search for features in the basemap. If there are layers in this map that you want users to interact with, move them to the Operational map.

    Note: The source data in both maps comes from The National Map https://viewer.nationalmap.gov/basic/

    Click on the Basemap tab to activate the view, and see that it is empty. We’ll make use of this map later.

    4. Activate the Operational map by clicking on its tab, click the Basemap gallery and choose the VTPK_Source_BM basemap

    Pan and zoom around the map to familiarize yourself with the basemap and operational data. This will give you context for how the operational layers work with the basemap layers.

    Note: if you do not see any labels, check an Operational layer off and back on to force a redraw.

    Create a Vector Tile Basemap

    The best practice for creating an efficient offline map is to use vector tiles in a Vector Tile Package (VTPK) for vector data and a Tile Package (TPK) for raster data. In this example, we’ll use vector tiles for the basemap.

    Note: Raster file data (imagery) won’t be covered in this example. A future update to this exercise will use it, but at this time raster file data in an offline map is not supported in apps built on the ArcGIS Runtime SDKs (including Explorer an d those built through AppSudio ). The teams are working to support raster files in offline maps in coming releases.

    You can use Tile Packages (.TPK) in Mobile Map Packages. You can use the Create Map Tile Package geoprocessing tool to create them.

    1. We could create an offline map of all of the content of the map however, it’s common to create an offline map or multiple offline maps from a portion of an existing map. For example, we may have a project including the entire United States, and we are only interested in making an offline map for a specific city. With that in mind we’ll make an offline map for a portion of this map.

    Zoom to the extent of Grand Teton National Park, this will be the extent of the offline map that we create.

    Activate the VTPK_Source_BM basemap. All of the maps in the project are linked by scale and extent, you should see the same extent of Grand Teton National park. The data in this project is comprised of vector file geodatabase data. We could use this data offline, as is, but the most efficient way to use vector data offline is in the form of vector tiles. This is especially true for our basemap, which does not need to provide access to attributes or user interaction with features. Vector tiles provide visualization and do not provide access to feature geometries or attributes. As mentioned earlier, if map users need access to attributes and geometries, add the layer to the Operational map. (If you look closely at this project you’ll notice that some of the same layers are in both the Operational map and VTPK_Source_BM basemap. This allows efficient feature visualization while still providing access to feature attributes.)

    2. The tools for creating a vector tile package will create vector tiles for the map’s defined extent. Let’s set the extent of the map to Grand Teton National Park by right-clicking on VTPK_Source_BM in the table of contents and clicking Properties . Click Extent and set the Custom extent to the Current visible extent and click OK .

    When you click the Full Extent button on the ribbon, the map now zooms to the custom extent. Use this button to confirm that what you are seeing in the view is what you are going to get when creating the Vector Tiles.

    Next, we ‘ll use a Geoprocessing tool to create vector tiles for the basemap.

    3. Click the Analysis tab and click the Tools button.

    This opens the Geoprocessing tools pane.

    4. Search for “Vector Tile Package” and open the Create Vector Tile Package tool.

    5. Choose VTPK_Source_BM for your input map.

    6. I n the Output File , specify a name and output location for your Vector Tile Package.

    7. Uncheck Package for ArcGIS Online | Bing Maps | Google Maps

    The tiling scheme automatically updates to match the coordinate system of the map.

    Note: The tiling scheme shows the coordinate system defined in the Map Properties. In this case, the coordinate system being used matches the coordinate system of the map and the data. To use the ArcGIS Online | Bing Maps | Google Maps scheme the map needs to be in the WGS 84 Web Mercator auxiliary sphere coordinate system.

    Note: You can change the coordinates system for each map in the project to WGS 84 Web Mercator auxiliary sphere and produce vector tiles in the ArcGIS Online | Bing Maps | Google Maps tiling scheme. However, drawing performance in ArcGIS Pro will be less efficient, as all of the map’s data is in Geographic Coordinate System North American 1983, and displaying the layer will require more processing as the data is re-projected on the fly. When possible, always put your map and data in the same coordinate system for the fastest data display.

    8. Accept the Default tiling format, which is Indexed .

    9. Set the Maximum Cached Scale to 0 . Some offline map use cases require zooming in close to the map, and without this setting the vector tiles stop displaying once past the maximum cached scale. By setting this to 0, users can zoom in as close as they want and still see vector tile information.

    10. The tool will use a default index, so Index Polygons are not needed. The Summary and Tags are also optional and not necessary to fill in.

    11. Before running the tool, right-click on the VTPK_Source_BM map in the Contents pane and click Properties .

    Click Metadata and write a Description for the map. If the map doesn’t have a description, the Create Vector Tile Package tool will fail to run.

    (Yes, this could have been done as part of the example project, but this is an often-experienced road block to success… so good to know!)

    Click OK to add the Description.

    12. Click Run on the Create Vector Tile Package tool.

    Using Vector Tile Package as a basemap

    1. When the tool has completed, click the Basemap tab to activate the empty Basemap view.

    Click Map Tab and Add Data , browse to the Vector Tile Package that you created, select it, and click OK .

    2. Click the Operational Tab to activate the Operational Map view.

    In this view, you’ll switch basemaps to use the “Basemap” that you just added the Vector Tile Package to.

    3. Choose the Basemap from the Basemap Gallery.

    The basemap in your Operational view updates to use the Vector Tile Package you created.

    4. Right-click on Basemap in the Contents pane and select Zoom to Layer .

    This is the extent of the map that we will be sharing for offline use.

    Let’s make the map searchable. There are two ways to provide search functionality for offline map users. We can enable feature search or create locators to use with the map (or both). One of the key differences is that locators support suggestions as the user is typing, which aren’t supported in feature search. The tradeoff is that locators increase the size of the map. Feature search doesn’t increase the size of the map, but it doesn’t provide suggestions.

    Note: Different apps support different types of search functionality. Both are supported in Explorer. Navigator for ArcGIS only supports locators. It is up to the developer with an AppStudio or Runtime SDK app.

    Let’s start by enabling feature search, and then we’ll create a locator.

    1. Click Locate on the Ribbon and click Settings on the Locate Pane.

    The online locators associated with your Organization are listed. These won’t be accessible for offline map users in your organization or users you share the map with who are not members of your organization, so you need to include search functionality in the map.

    2. Click the “Add Locate Provider” button and click Add Layer

    The feature layers in your map will be listed. Let’s allow users of the map to find mountain peaks. Highlight the TowerSummitMinesDam layer and click OK .

    Note: many of the operational layers in this map reference the same Geonames USA point data. A layer definition query is being used for cartographic purposes to group varying Geonames Point of Interest types

    This shows all Fields in this layer and allow us to choose a search operator for each field that we want to make available for search by users of the map.

    3. Allow map users to search by FEATURE_NAME and use the Contains This will return search results for all features in the TowerSummitMinesDam layer that “contain” the characters typed in by the map user. If you use the “Equals” operator the user will need to match the attribute exactly to find it.

    4. Click the Back Arrow to save and return to settings.

    5. The Layer is listed in the Locator list. Uncheck the locators provided by your organization and then click Locate to test the feature search on the TowerSummitMinesDam layer.

    6. Search for ‘Grand Te’ and see that Grand Teton is returned in the search results.

    If needed, you can enable search for more operational layers in your map and multiple fields in those layers.

    7. Another way to provide search capabilities for your Explorer users is to include a locator in the map. You can create locators for feature layers in your map by using the Create Address Locator geoprocessing tool.

    8. Choose a Locator Style, Gazetteer and Single Field work well for Named features. For this case use the Gazetteer style.

    9. Choose a layer to build the locator from. This feature layer does not need to be in your map. For this example, use the LocalePopPlace feature layer.

    10. Specify the attribute field with the attributes to search for: FEATURE_NAME .

    11. Specify an output address locator name or just use the default.

    12. Check Enable Suggestions . As users type their search string into the map, suggestions will be presented (which isn’t supported with feature search).

    13. Click Run to create the locator.

    In Locate settings you will see that the locator you just created is available for use.

    14. Search for “White Grass” and see “White Grass Ranch” and “White Grass Ranger Station” returned in the results.

    Bookmarks are predefined named extents that you can include in the map. Bookmarks allow map users to easily zoom to an extent. Bookmarks that you create for the map in ArcGIS Pro are included in the offline map.

    1. To create a bookmark, zoom to a specific extent, click Bookmarks on the ribbon, and click New Bookmark .


    Spatial Data in R

    This next part of the post applies the same principles introduced in earlier post to the much more complex problem of handling spatial data within R. In this article we will produce a gallery of maps using many of the plotting tools available in R. The resulting maps will not be that meaningful- the focus here is on sound visualisation with R and not sound analysis (I know one is useless without the other!). Good quality spatial analysis will come in the rest of the module.

    Whilst the instructions are step by step you are encouraged to start deviating from them (trying different colours for example) to get a better understanding of what we are doing.

    In this section we’ll require even more specialist packages, so I should probably spend some more time explaining what packages actually are! Packages are bits of code that extend R beyond the basic statistical functionality it was originally designed for. For spatial data, they enable R to process spatial data formats, carry out analysis tasks and create some of the maps that follow.

    Bascially, without packages, R would be very limited. With packages, you can do virtually anything! One of the issues you will come across is that packages are being continually developed and updated and unless you keep your version of R updated and your packages updated, there may be some functions and options not available to you. This can be a problem, particularly with University installations which (at best) may only get updated once a year. Therefore, apologies in advance if things don’t work as intended!

    Packages in RStudio

    1. In RStudio all packages can be installed and activated in the ‘Packages’ tab in the bottom- right hand window:

    As with everything else in R though, we can also run everything from the command line.The first package we need to install for this part of the practical is maptools — either find and install it using the RStudio GUI or do the following:

    There are a few other packages we’ll need to get to grips with. Some, like ggplot2 (one of the most influential R packages ever) are part of the tidyverse package we came across earlier. Others we will need to install for the first time.

    #install.packages(c(“classint”, “OpenStreetMap”, “tmap”))

    # — might also need these ones(“RColorBrewer”, “Sp”, “rgeos”, “tmap”, “tmap tools”, “sf”, “downloader”, “rgdal”, “geojsonio”)

    Now that the packages have been installed you will not have to repeat the above steps again (when you use your account in these cluster rooms). Open a new script and save it to your working directory as “wk3_maps.r”. As before, type each of the lines of code into this window and then select and use the ctrl return keys to run them. Be sure to save your script often.

    The first task is to load the packages we have just installed . note , you might have some issues with the OpenStreetMap package if your installation of java on your computer doesn’t match your installation of R — e.g. if you have installed the 64bit version of R, you also need the 64bit version of java (same with the 32bit versions) — you may also need to install the package Rcpp separately and try again.

    #Load Packages (ignore any error messages about being built under a #differ ent R version):
    library(maptools)library(RColorBrewer)library(classInt) library(OpenStreetMap) library(sp) library(rgeos) library(tmap)library(tmaptools) library(sf) library(rgdal) library(geojsonio)

    A Brief Background to Spatial Data in R

    R has a very well developed ecosystem of packages for working with Spatial Data. Early pioneers like Roger Bivand and Edzer Pebesma along with various colleagues were instrumental in writing packages to interface with some powerful open source libraries for working with spatial data, such as GDAL and GEOS. These were accessed via the rgdal and rgeos packages. The maptools package by Roger Bivand, amongst other things, allowed Shapefiles to be read into R. The sp package (along with spdep ) by Edzer Pebesma was very important for defining a series of classes and methods for spatial data natively in R which then allowed others to write software to work with these formats. Other packages like raster advanced the analysis of gridded spatial data, while packages like classint and RColorbrewer facilitated the binning of data and colouring of choropleth maps.

    Whilst these packages were extremely important for advancing spatial data analysis in R, they were not always the most straighforward to use — making a map in R could take quite a lot of effort and they were static and visually basic. However, more recently new packages have arrived to change this. Now leaflet enables R to interface with the leaflet javascript library for online, dynamic maps. ggplot2 which was developed by Hadley Wickam and colleagues radically changed the way that people thought about and created graphical objects in R, including maps, and introduced a graphical style which has been the envy of other software to the extent that there are now libraries in Python which copy the ggplot2 style!

    Building on all of these, the new tmap (Thematic Map) package has changed the game completely and now enables us to read, write and manipulate spatial data and produce visually impressive and interactive maps, very easily. In parallel, the sf (Simple Features) package is helping us re-think the way that spatial data can be stored and manipulated. It’s exciting times for geographic information / spatial data science!

    Making some choropleth maps

    Choropleth maps are thematic maps which colour areas according to some phenomenon. In our case, we are going to fill some irregular polygons (the London Boroughs) with a colour that corresponds to a particular attribute.

    As with all plots in R, there are multiple ways we can do this. The basic plot() function requires no data preparation but additional effort in colour selection/ adding the map key etc. qplot() and ggplot() (installed in the ggplot2 package) require some additional steps to format the spatial data but select colours and add keys etc automatically. Here, we are going to make use of the new tmap package which makes making maps very easy indeed.

    Reading a shape!le / any other spatial data source

    6. So one mega cool thing about R is you can read spatial data in straight from the internetz! Try this below for downloading a GeoJson file:

    EW < geojson_read (“http://geoportal.statistics.gov.uk/datasets/8edafbe3276 d4b56aec60991cbddda50_2.geojson”, what = “sp”)

    #pull out london using grep and the regex wildcard for’start of the string’ (^) to to look for the bit of the district code that relates to London (E09 ) from the ‘lad15cd’ column in the data slot of our spatial polygons datafr ame


    Geospatial Development By Example with Python

    Geospatial Development By Example with Python is intended for beginners or advanced developers in Python who want to work with geographic data. The book is suitable for professional developers who are new to geospatial development, for hobbyists, or for data scientists who want to move into some simple development.

    • Prepare a development environment with all the tools needed for geo-processing with Python
    • Import point data and structure an application using Python's resources
    • Combine point data from multiple sources, creating intuitive and functional representations of geographic objects
    • Filter data by coordinates or attributes easily using pure Python
    • Make press-quality and replicable maps from any data
    • Download, transform, and use remote sensing data in your maps
    • Make calculations to extract information from raster data and show the results on beautiful maps
    • Handle massive amounts of data with advanced processing techniques
    • Process huge satellite images in an efficient way
    • Optimize geo-processing times with parallel processing

    From Python programming good practices to the advanced use of analysis packages, this book teaches you how to write applications that will perform complex geoprocessing tasks that can be replicated and reused.

    Much more than simple scripts, you will write functions to import data, create Python classes that represent your features, and learn how to combine and filter them.

    With pluggable mechanisms, you will learn how to visualize data and the results of analysis in beautiful maps that can be batch-generated and embedded into documents or web pages.

    Finally, you will learn how to consume and process an enormous amount of data very efficiently by using advanced tools and modern computers' parallel processing capabilities.

    This easy-to-follow book is filled with hands-on examples that illustrate the construction of three sample applications of how to write reusable and interconnected Python code for geo-processing.


    Spatial Data in R

    This next part of the post applies the same principles introduced in earlier post to the much more complex problem of handling spatial data within R. In this article we will produce a gallery of maps using many of the plotting tools available in R. The resulting maps will not be that meaningful- the focus here is on sound visualisation with R and not sound analysis (I know one is useless without the other!). Good quality spatial analysis will come in the rest of the module.

    Whilst the instructions are step by step you are encouraged to start deviating from them (trying different colours for example) to get a better understanding of what we are doing.

    In this section we’ll require even more specialist packages, so I should probably spend some more time explaining what packages actually are! Packages are bits of code that extend R beyond the basic statistical functionality it was originally designed for. For spatial data, they enable R to process spatial data formats, carry out analysis tasks and create some of the maps that follow.

    Bascially, without packages, R would be very limited. With packages, you can do virtually anything! One of the issues you will come across is that packages are being continually developed and updated and unless you keep your version of R updated and your packages updated, there may be some functions and options not available to you. This can be a problem, particularly with University installations which (at best) may only get updated once a year. Therefore, apologies in advance if things don’t work as intended!

    Packages in RStudio

    1. In RStudio all packages can be installed and activated in the ‘Packages’ tab in the bottom- right hand window:

    As with everything else in R though, we can also run everything from the command line.The first package we need to install for this part of the practical is maptools — either find and install it using the RStudio GUI or do the following:

    There are a few other packages we’ll need to get to grips with. Some, like ggplot2 (one of the most influential R packages ever) are part of the tidyverse package we came across earlier. Others we will need to install for the first time.

    #install.packages(c(“classint”, “OpenStreetMap”, “tmap”))

    # — might also need these ones(“RColorBrewer”, “Sp”, “rgeos”, “tmap”, “tmap tools”, “sf”, “downloader”, “rgdal”, “geojsonio”)

    Now that the packages have been installed you will not have to repeat the above steps again (when you use your account in these cluster rooms). Open a new script and save it to your working directory as “wk3_maps.r”. As before, type each of the lines of code into this window and then select and use the ctrl return keys to run them. Be sure to save your script often.

    The first task is to load the packages we have just installed . note , you might have some issues with the OpenStreetMap package if your installation of java on your computer doesn’t match your installation of R — e.g. if you have installed the 64bit version of R, you also need the 64bit version of java (same with the 32bit versions) — you may also need to install the package Rcpp separately and try again.

    #Load Packages (ignore any error messages about being built under a #differ ent R version):
    library(maptools)library(RColorBrewer)library(classInt) library(OpenStreetMap) library(sp) library(rgeos) library(tmap)library(tmaptools) library(sf) library(rgdal) library(geojsonio)

    A Brief Background to Spatial Data in R

    R has a very well developed ecosystem of packages for working with Spatial Data. Early pioneers like Roger Bivand and Edzer Pebesma along with various colleagues were instrumental in writing packages to interface with some powerful open source libraries for working with spatial data, such as GDAL and GEOS. These were accessed via the rgdal and rgeos packages. The maptools package by Roger Bivand, amongst other things, allowed Shapefiles to be read into R. The sp package (along with spdep ) by Edzer Pebesma was very important for defining a series of classes and methods for spatial data natively in R which then allowed others to write software to work with these formats. Other packages like raster advanced the analysis of gridded spatial data, while packages like classint and RColorbrewer facilitated the binning of data and colouring of choropleth maps.

    Whilst these packages were extremely important for advancing spatial data analysis in R, they were not always the most straighforward to use — making a map in R could take quite a lot of effort and they were static and visually basic. However, more recently new packages have arrived to change this. Now leaflet enables R to interface with the leaflet javascript library for online, dynamic maps. ggplot2 which was developed by Hadley Wickam and colleagues radically changed the way that people thought about and created graphical objects in R, including maps, and introduced a graphical style which has been the envy of other software to the extent that there are now libraries in Python which copy the ggplot2 style!

    Building on all of these, the new tmap (Thematic Map) package has changed the game completely and now enables us to read, write and manipulate spatial data and produce visually impressive and interactive maps, very easily. In parallel, the sf (Simple Features) package is helping us re-think the way that spatial data can be stored and manipulated. It’s exciting times for geographic information / spatial data science!

    Making some choropleth maps

    Choropleth maps are thematic maps which colour areas according to some phenomenon. In our case, we are going to fill some irregular polygons (the London Boroughs) with a colour that corresponds to a particular attribute.

    As with all plots in R, there are multiple ways we can do this. The basic plot() function requires no data preparation but additional effort in colour selection/ adding the map key etc. qplot() and ggplot() (installed in the ggplot2 package) require some additional steps to format the spatial data but select colours and add keys etc automatically. Here, we are going to make use of the new tmap package which makes making maps very easy indeed.

    Reading a shape!le / any other spatial data source

    6. So one mega cool thing about R is you can read spatial data in straight from the internetz! Try this below for downloading a GeoJson file:

    EW < geojson_read (“http://geoportal.statistics.gov.uk/datasets/8edafbe3276 d4b56aec60991cbddda50_2.geojson”, what = “sp”)

    #pull out london using grep and the regex wildcard for’start of the string’ (^) to to look for the bit of the district code that relates to London (E09 ) from the ‘lad15cd’ column in the data slot of our spatial polygons datafr ame


    The Development of National Datasets

    Software advancements were certainly a large piece of the digital-mapping puzzle, but without readily available data, progress in this arena would have been slowed. Fortunately, a few agencies within the US federal government had vested interest and made headway. Two of the front-runners were the USCB and the USGS , and their timing was fitting.

    US Census Bureau

    (USCB Logo for the United States Census Bureau’s TIGER (Topologically Integrated Geographic Encoding and Referencing) map data format

    These early maps began as hand sketches created by field surveyors and included features such as civil divisions, public works, hydrography, and hypsography, but by the 1930s the USGS started using aerial photography and photogrammetry.

    USGS 7.5 Minute Map (HowToWilderness.com Aerial Photo Stowe VT)

    Overlaying the Stories

    The motivation for GIS has varied widely, but most of the field’s innovation grew from academic curiosity and the governmental challenges of looking for solutions with large datasets (Coppock and Rhind 1991, 21–43).

    When you explore the individual stories of GIS history, you see that chance, timing, and heritage played into its development.

    You see how the British, with their obsession with surveying and mapping, laid much of the groundwork for GIS and that many of the early key players in mapping were of British heritage. Not only did they have access to accurate maps and tools — they were also wired with the mindset that mapping advances society.

    Yet, it took the composition of North America to give rise to the first GIS. The abundance of public land in Canada and the United States (which Britain lacked) created the need for complex land management systems. You see that by the early 1980s, there were more than one thousand GIS systems in North America alone, which, proportionately, was extremely high (Tomlinson 1985).

    But development of the first GIS also took a bit of chance. It wasn’t enough for the British-born Roger Tomlinson to have conceived the concept it also took him sitting in the right seat on the right plane (i.e., next to Lee Pratt) on the right day to close the final gap.

    The first generation of GIS was dominated by mainframe computers and punch card technology. You see the reoccurrence of the IBM System 360 in the 1960s and its ability to be programmed to cover a range of applications.

    You also see how timing played a role in the second generation of GIS, as personal computing and software applications were the focus. Jack Dangermond’s timing at the Lab put him in position to successfully launch Esri, and then, by the 1980s, personal computers were prevalent enough for Esri to bring the first cross-platform, do-it-yourself GIS application to market.

    The current GIS platform is built on web applications, shareable web services, and cloud computing. But, in this article we explored the interwoven stories that you discover when you zoom in on the history of GIS. Hopefully, when you zoom back out you will now have a greater appreciation for today’s GIS.

    Key moments in GIS. Click for larger version.

    Note: a special thanks to the editor, Anna Tribolet for her awesome editing skills and for the creation of the ‘Key Moments in GIS’ infographic!

    Article republished with permission from Utah AGRC.

    References

    Aguirre, Jessica Camille. “The Unlikely History of the Origins of Modern Maps.” Smithsonian.com. June 2, 2014. Accessed August 19, 2019. https://www.smithsonianmag.com/history/unlikely-history-origins-modern-maps-180951617/.

    Anam, Aysha. “How India was measured: Story of the Great Trigonometrical Survey of India.” India Today. Published July 10, 2018. Updated July 12, 2018. Accessed August 19, 2019. https://www.indiatoday.in/education-today/gk-current-affairs/story/how-was-india-measured-why-is-it-important-to-measure-land-1281835-2018-07-10.

    ArcGIS. 1986 PC ARC/INFO Released. Retrieved from “Our Common History” story map. https://www.arcgis.com/apps/MapJournal/index.html?appid=df8224427fc741ab9f8117e9af1d692a.

    Artz, Matt. “Charting the Unknown: How Computer Mapping at Harvard Became GIS.” GIS and Science. November 17, 2009. Accessed August 19, 2019. https://gisandscience.com/2009/11/17/charting-the-unknown-how-computer-mapping-at-harvard-became-gis/.

    Beniger, James Ralph. The Control Revolution: Technological and Economic Origins of the Information Society, 411–412. Cambridge (MA): Harvard University Press, 1997. Accessed on 7/31/2019 via Google Books.

    Chrisman, Nicholas. “Remembering the Lab: A short history of GIS at Harvard.” Presentation. 2006. Accessed August 19, 2019. https://docplayer.net/12667857-Remembering-the-lab.html.

    Chrisman, Nick. Charting the Unknown: How Computer Mapping at Harvard Became GIS. Redlands, CA: Esri Press, 2006.

    Coppock, J. Terry, and David W. Rhind. “The History of GIS.” In Applications. Vol. 2 of Geographical Information Systems: Principles and Applications. Edited by David J. Maguire,

    Michael F. Goodchild, and David W. Rhind, 21–43. Harlow, UK: Longmans, 1991. https://oxfordbibliographiesonline.com/view/document/obo-9780199874002/obo-9780199874002-0143.xml#obo-9780199874002-0143-bibItemGroup-0001.

    DeMers, Michael. “CGIS History Captioned.” Youtube. Uploaded by Michael DeMers on August 29, 2013. Accessed August 19, 2019. https://www.youtube.com/watch?v=3VLGvWEuZxI.

    DeMers, Michael. CGIS Screenshot. 2013. Screenshot from “CGIS History Captioned” YouTube video. https://www.youtube.com/watch?v=3VLGvWEuZxI.

    Esri. “History of GIS.” Accessed August 19, 2019. https://www.Esri.com/en-us/what-is-gis/history-of-gis.

    Esri. “History Up Close.” 2015. Accessed August 19, 2019. https://www.Esri.com/

    Fisk, Dale. “Programming with Punched Cards.” Columbia University. Accessed August 19, 2019. http://www.columbia.edu/cu/computinghistory/fisk.pdf.

    Friendly, M. & Denis, D. J. Early SYMAP image of Connecticut. 2001. “1950-1974: Re-birth of data visualization” article on Milestones in the history of thematic cartography, statistical graphics, and data visualization site. http://www.datavis.ca/milestones/. Accessed August 29, 2019.

    Garfield, Simon. On the Map: A Mind-Expanding Exploration of the Way the World Looks, 192, 194. New York: Gotham Books, 2013.

    GIS Geography. “The Remarkable History of GIS.” June 13, 2019. Accessed August 19, 2019. https://gisgeography.com/history-of-gis/.

    GISHistory. “Allan Schmidt talks about the Harvard Laboratory for Computer Graphics and Spatial Analysis, Part 1.” Youtube. Uploaded on November 2, 2009. Accessed August 19, 2019. https://www.youtube.com/watch?v=B14LfcJN490&feature=youtu.be.

    GISHistory. “Jack Dangermond talks about the Harvard Laboratory for Computer Graphics and Spatial Analysis.” Youtube. Uploaded on October 30, 2009. Accessed August 19, 2019. https://www.youtube.com/watch?v=BaHXDMid78I&feature=youtu.be.

    HowToWilderness.com. Aerial Photo Stowe VT. Retrieved from “Reading a Map” article. https://howtowilderness.com/map-training/.

    Jones, Douglas W. “Punched Cards: A brief illustrated technical history.” The University of Iowa Department of Computer Science. Last modified June 28, 2017. Accessed August 19, 2019. http://homepage.divms.uiowa.edu/

    Library of Congress. Image of punched card of Herman Hollerith. 1895. Retrieved from Wikipedia Commons image “File:Hollerith Punched Card.jpg.” http://memory.loc.gov/mss/mcc/023/0008.jpg.

    Micronautes. Broad Street Map/Broadwick Street Water Pump. 2017. Retrieved from “John Snow and cholera” article by Laura Gómez. http://unautes.com/index.php/2017/11/16/john-snow-and-cholera/?lang=en.

    Ordnance Survey. “Our History.” 2019. Accessed August 19, 2019. https://www.ordnancesurvey.co.uk/about/overview/history.html.

    Parecki, Aaron. Esri in 1969. Jack Dangermond, Scott Morehouse, SJ Camarata, Logan Hardison. 2013. Retrieved from Flickr. https://www.flickr.com/photos/aaronpk/9570322487/in/photostream/.

    Pascual, Daniel Fernández. Great Trigonometrical Survey of India. 2012. Retrieved from “Mapping an Empire” article on Deconcrete.org. https://www.deconcrete.org/2012/03/12/mapping-an-empire/.

    Poiker, Thomas K., and Ian K. Crain. “Geographic Information Systems.” In The Canadian Encyclopedia. Historica Canada. Article published February 07, 2006 Last edited February 3, 2014. Accessed August 19, 2019. https://www.thecanadianencyclopedia.ca/en/article/geographic-information-systems

    Rabbitt, Mary C. “The United States Geological Survey: 1879-1989.” Circular, 1989. https://pubs.usgs.gov/circ/1050/pdf/CIRC1050.pdf.

    Schuster, Adam. Proto IBM. 2007. Retrieved from Flickr. https://www.flickr.com/photos/[email protected]/411109339.

    Thompson, Clive. “From Ptolemy to GPS, the Brief History of Maps.” Smithsonian Magazine. July 2017. Accessed August 19, 2019. https://www.smithsonianmag.com/innovation/brief-history-maps-180963685/.

    Tomlinson 1984: Geographic Information Systems – the new frontier. The Operational Geographer 5: 31-6.

    Tomlinson, Roger. “Origins of the Canada Geographic Information System.” Esri. 2012. Accessed August 19, 2019. https://www.Esri.com/news/arcnews/fall12articles/origins-of-the-canada-geographic-information-system.html.

    University Consortium for Geographic Information Science. “Roger Tomlinson.” Accessed August 19, 2019. https://www.ucgis.org/roger-tomlinson.

    US Census Bureau.“Agency History.” Last revised May 30, 2019. Accessed August 19, 2019. https://www.census.gov/history/www/census_then_now/.

    US Census Bureau. “Census Bulletin.” Published September 6, 1968. Accessed August 19, 2019. https://www.census.gov/history/pdf/1968censusbulletin-dime.pdf.

    US Census Bureau. “Dual Independent Map Encoding.” Last revised May 30, 2019. Accessed August 19, 2019. https://www.census.gov/history/www/innovations/technology/dual_independent_map_encoding.html.

    US Census Bureau. “Herman Hollerith.” Last revised May 17, 2018. Accessed August 19, 2019. https://www.census.gov/history/www/census_then_now/notable_alumni/herman_hollerith.html.

    US Geological Survey. “National Geospatial Program.” Accessed August 29, 2019. https://www.usgs.gov/core-science-systems/national-geospatial-program/topographic-maps.

    US Geological Survey. The National Map. Last modified October 22, 2018. https://viewer.nationalmap.gov/advanced-viewer/.

    Waldheim, Charles. “The Invention of GIS.” The Harvard Gazette. Video/Production: Ned Brown. October 12, 2011. Accessed August 19, 2019. https://news.harvard.edu/gazette/story/2011/10/the-invention-of-gis/.

    Winfields Outdoors. Image 1 from “Interview With Ordnance Survey – History & Future of OS Maps” article. February 18, 2016. Accessed August 29, 2019. https://www.winfieldsoutdoors.co.uk/blog/ordnance-survey-interview/.