[comment]: # (Set the theme:) [comment]: # (The list of themes is at https://revealjs.com/themes/) [comment]: # (The list of code themes is at https://highlightjs.org/) [comment]: # (Pass optional settings to reveal.js:) [comment]: # (markdown: { smartypants: true }) [comment]: # (plugins:[ RevealHighlight ]) [comment]: # (Other settings are documented at https://revealjs.com/config/) # GIS Analyses using Free and Open Source Software Christina Ludwig, Veit Ulrich November 20, 2024 | Institute of Geography | Heidelberg University

# Agenda 0. Course Organisation 1. Open (Spatial) Data 2. Exercise: Automatically downloading data 3. Group project

# 0. Course Organisation

## 1. Recap Assignment 1

### ChatGPT We do not encourage the use of ChatGPT, because we think that you will learn the methods better if you come up with the solutions yourself.

2. Find three different **FOSSGIS** tools to calculate the TWI. Note not just name of the software (e.g. QGIS) but the name of the specific tool. At least one of them should **not** be included in the QGIS Processing Toolbox. - Be careful to name the tools correctly - The Whitebox tool is called "Wetness index" - Some groups did not mention the correct name of the tool (no minus points)

### CRS Issues Do not use project CRS in model to make it possible to run on other projects (reproducibility) <img src="./img/as01_reproject_use-project-crs.png" width=1200px />

### CRS Issues - Target CRS: "Use project CRS" should be unticked, define the target CRS - In some cases, a user-defined CRS was used, but later overwritten with a hardcoded CRS - If there is CRS input, use it and not set it later to another CRS statically without the user knowing <img src="./img/as01_crs_hardcoded.PNG" width=1200px />

### GRASS TWI vs. SAGA TWI The value ranges of TWIs derived with different TWI tools may differ. <img src="./img/grass_twi.png" width=700px, align=left /> <img src="./img/saga_twi.png" width=700px, align=right />

### Figures What could be improved in this figure? <img src="./img/map_flood.png" width=1000px />

- Improve legend: It is not clear what level of flooding of 0 and 1 means - Clearly delineate AOI border - Add "OpenStreetMap contributors" to the OSM citation - Fix typos - Flooded areas are much bigger than in the observed_event.geojson (Ok, because from the assignment it was not clear if one of the thresholds 5, 10, and 15 should be used or if any threshold could be used) <img src="./img/map_flood.png" width=900px />

### Figures What could be improved in this figure? <img src="./img/map_twi.jpg" width=1000px />

### Figures - Clearly delineate AOI border - Clip TWI map to the AOI - Add basemap <img src="./img/map_twi.jpg" width=1000px, align=right />

### Figures - good examples (partly subjective) <img src="./img/map_flood_nice.jpg" width=1000px />

### Figures - good examples (partly subjective) <img src="./img/map_twi_nice.png" width=1000px />

### Other - Clip raster to AOI before processing to save computing time (and energy) - Interpretation of results is as important as data processing

## 2. Group Project Information on the [course website](http://fossgis.courses-pages.gistools.geog.uni-heidelberg.de/home/content/group_project/intro.html) - Group project discussions on 2.12. and 4.12. - Your project proposal must be finished until then! - Who has time on Monday, 2.12. in the morning?

# 1. Open Data

## Discuss in groups (15 minutes) 1. What is required so that a data set can be called **'open'**? 2. From a user perspective, what would be **suitable file formats** to publish a vector data set online? **Why?** (e.g. pdf, xlsx, csv, geojson, shapefile) 3. What are different ways to **download** spatial data sets or **view them in QGIS**?

## 1. What is required so that a data set can be called 'open'?

## The Open Definition by The Open Knowledge Foundation An open work (e.g. piece of knowledge or data) must satisfy the following requirements in its distribution: 1. **Access:** The work must be provided as a whole and at no more than a reasonable one-time reproduction cost, and should be downloadable via the Internet without charge. 2. **Machine Readability:** The work must be provided in a form readily processable by a computer easy to access and use 3. **Open Format:** The work must be provided in an open format (non proprietary), e.g., xml 4. **Open License or Status:** The work must be in the public domain (absence of copyright or restrictions) or provided under an open license. Source: [https://opendefinition.org/od/2.1/en/](https://opendefinition.org/od/2.1/en/)

## 2. From a user perspective, what would be **suitable file formats** to publish a vector data set online? **Why?** - pdf - xlsx (excel) - csv - geojson - shapefile

## 5-star deployment scheme for Open Data as suggested by Tim Berners-Lee, the inventor of the Web and Linked Data initiator. <img src="./img/5-star-steps.png" width=1000px /> - RDF: language for modelling metadata of online resources, facilitating data exchange without information loss, e.g., reusing or bookmarking data - Online information such as author, title, etc. is identified with uniform resource identifiers (URI) Source: [https://5stardata.info/en/](https://5stardata.info/en/)

## Open Data is growing Try [Google's Data Search Engine](https://datasetsearch.research.google.com/). A small collection of open geospatial data sources is provided on the [course webpage](http://fossgis.courses-pages.gistools.geog.uni-heidelberg.de/home/content/topics/07_apis/opendata.html).

## 3. What are different ways to download spatial data sets or view them in QGIS?

- Download file through web browser - WMS: [WebMapService](https://docs.qgis.org/3.34/en/docs/server_manual/services/wms.html) - WFS: [WebFeatureService](https://docs.qgis.org/3.34/en/docs/server_manual/services/wfs.html) - WMTS: [WebMapTileService](https://docs.qgis.org/3.34/en/docs/server_manual/services/wmts.html): uses cached image tiles - WCS: [WebCoverageService](https://docs.qgis.org/3.34/en/docs/server_manual/services/wcs.html): raster data that can be used as input for analysis - [XYZ Tile service](https://docs.qgis.org/3.34/en/docs/user_manual/managing_data_source/opening_data.html#using-xyz-tile-services), e.g. OSM - API: [Application Programming Interface](https://en.wikipedia.org/wiki/API) Video Tutorial: [Using WMS / WMTS / WFS in QGIS by Digital Geography](https://www.youtube.com/watch?v=LOHfnZBt_OQ)

## Exercise: Explore WFS and WMS layers in QGIS Do the WFS/WMS exercise on the [course webpage](http://fossgis.courses-pages.gistools.geog.uni-heidelberg.de/home/content/topics/07_apis/wfs_wms.html).

# 2. Downloading data using APIs ### Example: OpenStreetMap (OSM)

## There are many ways to download OSM data. e.g. - [Geofabrik Download Server](https://download.geofabrik.de/): download data region-wise - [https://overpass-turbo.eu/](https://overpass-turbo.eu/): download specific data - [Overpass API](https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL) - more options on [https://wiki.openstreetmap.org/wiki/Downloading_data](https://wiki.openstreetmap.org/wiki/Downloading_data)

## The ohsome API - API to download and analyse OSM data and its history - Developed by - [GIScience Research Group at Heidelberg University](https://www.geog.uni-heidelberg.de/gis/index_en.html) - [Heidelberg Institute for Geoinformation Technology (HeiGIT)](https://heigit.org/) - Open source project on https://github.com/GIScience/ohsome-api <p align="center"> <img src="./img/ohsome_api.png" width=1000px/> </p>

## ohsome: Downloading OSM features <img src="./img/trees.jpg" width=1000px />

## ohsome: History of OSM features <img src="./img/buildings_addrcity-not-null.gif" width=700px /> Temporal evolution of buildings with (green) and without address (red) in OSM within Heidelberg. [Learn how to create this animation here](http://k1z.blog.uni-heidelberg.de/2018/12/14/how-to-become-ohsome-part-1-visualizing-the-historical-evolution-of-osm-buildings-of-your-city/)

## ohsome API: Number of active OSM users <img src="./img/HD_ways.gif" width=900px />

## How to send a request to the ohsome API - **Endpoints** are different types of requests, e.g. - Download OSM features with geometry: https://api.ohsome.org/v1/elements/geometry - Download features with centroids: https://api.ohsome.org/v1/elements/centroid - Count features within an area: https://api.ohsome.org/v1/elements/count See documentation for full list of endpoints on [Data Extraction](https://api.ohsome.org/v1/swagger-ui.html?urls.primaryName=Data%20Extraction) and [Data Aggregation](https://api.ohsome.org/v1/swagger-ui.html?urls.primaryName=Data%20Aggregation). - **Parameters** specify the request, e.g. [time](https://docs.ohsome.org/ohsome-api/v1/time.html), [spatial boundary](https://docs.ohsome.org/ohsome-api/v1/boundaries.html), [OSM tags](https://docs.ohsome.org/ohsome-api/v1/filter.html)

## Example ### Query all trees in Heidelberg's Bahnstadt on 1. Oct 2024 #### Endpoint: https://api.ohsome.org/v1/elements/geometry #### Parameters - [Spatial boundary](https://docs.ohsome.org/ohsome-api/v1/boundaries.html): bboxes=8.65044,49.39118,8.68794,49.40992 - [OSM Tag Filter](https://docs.ohsome.org/ohsome-api/v1/filter.html): filter=natural=tree - [Time](https://docs.ohsome.org/ohsome-api/v1/time.html): time=2024-10-01

### Send as POST Request (recommended) Requires a program to send requests e.g., cUrl in the command line ``` curl -X POST \ --data-urlencode "bboxes=8.65443,49.39444,8.68533,49.40824" \ --data-urlencode "time=2024-10-01" \ --data-urlencode "filter=natural=tree" \ -o trees_2024.geojson \ "https://api.ohsome.org/v1/elements/geometry" ```

### Trees in Heidelberg Bahnstadt mapped in OSM up until 1. Nov 2021 <img src="./img/trees.jpg" width=1000px/>

### Alternative: Send as GET Request The same query can be sent as a GET request: **Query URL: {Endpoint}?{parameter}&{parameter}...** **Example:** [https://api.ohsome.org/v1/elements/geometry?bboxes=8.65443,49.39444,8.68533,49.40824&filter=natural=tree&time=2024-10-01](https://api.ohsome.org/v1/elements/geometry?bboxes=8.65443,49.39444,8.68533,49.40824&filter=natural=tree&time=2024-10-01)

## Exercise: Downloading data using APIs I've provided some **exercises** to practice downloading data using APIs (ohsome API and emissions API). It is available on the [course webpage](http://fossgis.courses-pages.gistools.geog.uni-heidelberg.de/home/content/topics/07_apis/api_exercise.html).