Shippensburg Land Projections: Ingest Layers for Visualization

[rajadain]

Ingest ProbabilityUrban_CentersOSI.tif and ProbabilityUrban_CorridorsOSI.tif for visualization only. These layers look like this:

(where the pink overlay is the DRB outline).

Use the same color ramp as used for past Shippensburg layers.

[rajadain]

My current approach is to use GDAL instead of our usual GeoTrellis based pipeline for this conversion. After some trial and error, I've settled on this toolchain maketiles.sh:

#!/bin/bash

SOURCE=$1
NAME="${SOURCE%.*}"

mkdir -p "output/$NAME-tiles"

pushd output

# Convert the input to 3857
gdalwarp -co TILED=YES -co COMPRESS=DEFLATE -t_srs EPSG:3857 ../"$NAME".tif "$NAME"_3857.tif
# Colorize with ramp
gdaldem color-relief -alpha -of PNG "$NAME"_3857.tif ../color-relief.txt "$NAME"_3857_colored.tif
# Create the tiles
gdal2tiles.py-3.8 -z 0-13 -r near -v -e --xyz "$NAME"_3857_colored.tif "$NAME-tiles"/

popd output

Where color-relief.txt is:

nv 255 255 255 0
0 255 255 255
0.13 246 226 226
0.26 237 196 196
0.39 227 167 167
0.52 218 137 137
0.65 209 107 107
0.78 199 78 78
0.9 190 50 50
1 183 28 28

This color ramp was made in QGIS using the same colors as used for the original Shippensburg layers. The first line makes NODATA values transparent.

On my local, this took 11m 11s to run this for one layer.

I'm going to make an instance in EC2 and do this there for the 6 layers.

[rajadain]

All the layers have been ingested, and are now available on S3:

❯ aws s3 ls tiles.us-east-1.azavea.com/ | grep shippensburg
    PRE shippensburg-2050-baseline-30m/
    PRE shippensburg-2050-scenario1-30m/
    PRE shippensburg-2050-scenario2-30m/
    PRE shippensburg-2100-centers-30m/
    PRE shippensburg-2100-centers-np-30m/
    PRE shippensburg-2100-centers-osi-30m/
    PRE shippensburg-2100-corridors-30m/
    PRE shippensburg-2100-corridors-np-30m/
    PRE shippensburg-2100-corridors-osi-30m/

The shippensburg-2100 are the new ones.

We were originally given a set of 6 tifs (original data is in the fileshare as shippensburg/ProbabilityMaps.zip):

❯ ll *.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:18 ProbabilityUrban_Centers.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:18 ProbabilityUrban_CentersNP.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:19 ProbabilityUrban_CentersOSI.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:19 ProbabilityUrban_Corridors.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:19 ProbabilityUrban_CorridorsNP.tif
-rw-r--r--@ 1 ttuhinanshu  1310893680   508M May  8 17:19 ProbabilityUrban_CorridorsOSI.tif

Each of these was reprojected to EPSG:3857 Web Mercator:

❯ gdalwarp -co TILED=YES -co COMPRESS=DEFLATE -t_srs EPSG:3857 ProbabilityUrban_CentersOSI.tif gdal/ProbabilityUrban_CentersOSI_3857.tif

which significantly reduced the file size:

❯ ll *_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    30M May 20 23:15 ProbabilityUrban_CentersNP_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    26M May 20 23:16 ProbabilityUrban_CentersOSI_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    26M May 20 17:05 ProbabilityUrban_Centers_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    36M May 20 23:17 ProbabilityUrban_CorridorsNP_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    31M May 20 23:17 ProbabilityUrban_CorridorsOSI_3857.tif
-rw-r--r--  1 ttuhinanshu  1310893680    31M May 20 23:16 ProbabilityUrban_Corridors_3857.tif

Then, an ingest EC2 instance was created in the azavea-datahub account, configured with default specs, all of which were free-tier eligible. docker and awscli were installed on the instance, and I added my azavea-datahub credentials.

Then the files were copied up to EC2:

❯ scp *_3857.tif ingest:~/data/
ProbabilityUrban_Centers_3857.tif               100%   26MB 696.8KB/s   00:38    
ProbabilityUrban_CentersNP_3857.tif             100%   30MB 690.0KB/s   00:44    
ProbabilityUrban_CentersOSI_3857.tif            100%   26MB 695.4KB/s   00:38    
ProbabilityUrban_Corridors_3857.tif             100%   31MB 688.3KB/s   00:45    
ProbabilityUrban_CorridorsNP_3857.tif           100%   36MB 696.8KB/s   00:53    
ProbabilityUrban_CorridorsOSI_3857.tif          100%   31MB 697.1KB/s   00:44    

The color relief configuration was also copied:

❯ scp color-relief.txt ingest:~/data/
color-relief.txt                                100%  157     6.3KB/s   00:00    

Then we colorized each TIF:

$ sudo docker run -v $(pwd)/data:/data osgeo/gdal /bin/bash -c "cd /data/; gdaldem color-relief -alpha -of PNG ProbabilityUrban_CentersNP_3857.tif color-relief.txt ProbabilityUrban_CentersNP_3857_colored.tif"

and then made tiles for zoom levels 0 to 13 for each TIF:

$ sudo docker run -v $(pwd)/data:/data osgeo/gdal /bin/bash -c "cd /data/; gdal2tiles.py -z 0-13 -r near -v -e --xyz ProbabilityUrban_CentersNP_3857_colored.tif ProbabilityUrban_CentersNP-tiles/"

Finally, the tiles were moved to S3:

$ aws s3 cp ProbabilityUrban_Centers-tiles s3://tiles.us-east-1.azavea.com/shippensburg-2100-centers-30m --recursive --acl public-read --exclude '*.html' --exclude '*.xml'

[rajadain]

The ingest AWS EC2 instance has been deleted.