Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GWLF-E Endpoint to Geoprocessing API #3483

Merged
merged 5 commits into from
Feb 23, 2022
Merged

Add GWLF-E Endpoint to Geoprocessing API #3483

merged 5 commits into from
Feb 23, 2022

Conversation

rajadain
Copy link
Member

@rajadain rajadain commented Feb 16, 2022 •
edited by emilyhu0106
Loading

Overview

Adds a public endpoint to run GWLF-E via the geoprocessing API. The input is the output of the endpoint added in #3480, either the result JSON or the job_uuid. In case the job is not ready or has failed, this returns an error.

This builds upon the work done in #3480.

Connects #3473

Demo

xh --verbose :8000/api/modeling/gwlf-e/prepare/ Authorization:"Token b0c671e1424f37e58c81670ce8118e8eff0f0c14" wkaoi=huc12__55174
POST /api/modeling/gwlf-e/prepare/ HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate, br
Authorization: Token b0c671e1424f37e58c81670ce8118e8eff0f0c14
Connection: keep-alive
Content-Length: 24
Content-Type: application/json
Host: localhost:8000
User-Agent: xh/0.13.0

{
    "wkaoi": "huc12__55174"
}



HTTP/1.1 200 OK
Allow: POST, OPTIONS
Connection: keep-alive
Content-Type: application/json
Date: Wed, 16 Feb 2022 20:32:39 GMT
Location: /api/jobs/ba67198e-4d43-4675-b308-3d1e21f8779f/
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept, Cookie, Origin

{
    "job": "ba67198e-4d43-4675-b308-3d1e21f8779f",
    "status": "started"
}
xh --verbose :8000/api/modeling/gwlf-e/run/ Authorization:"Token b0c671e1424f37e58c81670ce8118e8eff0f0c14" job_uuid=ba67198e-4d43-4675-b308-3d1e21f8779f
POST /api/modeling/gwlf-e/run/ HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate, br
Authorization: Token b0c671e1424f37e58c81670ce8118e8eff0f0c14
Connection: keep-alive
Content-Length: 49
Content-Type: application/json
Host: localhost:8000
User-Agent: xh/0.13.0

{
    "job_uuid": "ba67198e-4d43-4675-b308-3d1e21f8779f"
}



HTTP/1.1 200 OK
Allow: POST, OPTIONS
Connection: keep-alive
Content-Type: application/json
Date: Wed, 16 Feb 2022 20:33:12 GMT
Location: /api/jobs/eb1a233f-b52f-41e5-bc8f-2e8912d831f0/
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept, Cookie, Origin

{
    "job": "eb1a233f-b52f-41e5-bc8f-2e8912d831f0",
    "status": "started"
}
xh --verbose :8000/api/jobs/eb1a233f-b52f-41e5-bc8f-2e8912d831f0/ Authorization:"Token b0c671e1424f37e58c81670ce8118e8eff0f0c14"
GET /api/jobs/eb1a233f-b52f-41e5-bc8f-2e8912d831f0/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Authorization: Token b0c671e1424f37e58c81670ce8118e8eff0f0c14
Connection: keep-alive
Host: localhost:8000
User-Agent: xh/0.13.0

HTTP/1.1 200 OK
Allow: OPTIONS, GET
Connection: keep-alive
Content-Type: application/json
Date: Wed, 16 Feb 2022 20:33:44 GMT
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept, Cookie, Origin

{
    "job_uuid": "eb1a233f-b52f-41e5-bc8f-2e8912d831f0",
    "status": "complete",
    "result": {
        "meta": {
            "NYrs": 30,
            "NRur": 10,
            "NUrb": 6,
            "NLU": 16,
            "SedDelivRatio": 0.11972374125279198,
            "WxYrBeg": 1961,
            "WxYrEnd": 1990
        },
        "AreaTotal": 8210.8,
        "MeanFlow": 64849251.16010487,
        "MeanFlowPerSecond": 2.0563562645898297,
        "monthly": [
            {
                "AvPrecipitation": 7.924799999999999,
                "AvEvapoTrans": 0.5501632983146786,
                "AvGroundWater": 3.918457483654395,
                "AvRunoff": 3.7172997063186015,
                "AvStreamFlow": 8.878720710082073,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 7.248736666666668,
                "AvEvapoTrans": 0.8376727524299129,
                "AvGroundWater": 3.418162634169156,
                "AvRunoff": 3.6944916758425634,
                "AvStreamFlow": 8.235331037852175,
                "AvPtSrcFlow": 1.1226767278404555,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 8.751993333333331,
                "AvEvapoTrans": 2.4215409599807347,
                "AvGroundWater": 3.9516276188257184,
                "AvRunoff": 3.6500011554150147,
                "AvStreamFlow": 8.844592294349809,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 8.906509999999999,
                "AvEvapoTrans": 4.122753658162977,
                "AvGroundWater": 4.300639718909367,
                "AvRunoff": 1.020012441590038,
                "AvStreamFlow": 6.523520083185607,
                "AvPtSrcFlow": 1.202867922686202,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 9.634643333333335,
                "AvEvapoTrans": 6.070793901590392,
                "AvGroundWater": 3.749745873746373,
                "AvRunoff": 0.9920148097663717,
                "AvStreamFlow": 5.98472420362182,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 9.25576,
                "AvEvapoTrans": 6.396363367760513,
                "AvGroundWater": 2.662416345144882,
                "AvRunoff": 1.0373856266801131,
                "AvStreamFlow": 4.902669894511197,
                "AvPtSrcFlow": 1.202867922686202,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 10.811933333333334,
                "AvEvapoTrans": 6.834767614705511,
                "AvGroundWater": 2.583556299501228,
                "AvRunoff": 1.5279533829646463,
                "AvStreamFlow": 5.354473202574949,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 9.152890000000001,
                "AvEvapoTrans": 5.979568702089507,
                "AvGroundWater": 2.4012805866091154,
                "AvRunoff": 1.2051642956723883,
                "AvStreamFlow": 4.849408402390579,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 8.694420000000003,
                "AvEvapoTrans": 3.9978006163378006,
                "AvGroundWater": 2.445976436344594,
                "AvRunoff": 1.1965839755272936,
                "AvStreamFlow": 4.84542833455809,
                "AvPtSrcFlow": 1.202867922686202,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 6.980766666666666,
                "AvEvapoTrans": 3.231859369412581,
                "AvGroundWater": 2.754100166016227,
                "AvRunoff": 1.550724286202872,
                "AvStreamFlow": 5.547787972328175,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 8.39724,
                "AvEvapoTrans": 1.9477099778664069,
                "AvGroundWater": 2.980988871541816,
                "AvRunoff": 2.321118427037978,
                "AvStreamFlow": 6.504975221265996,
                "AvPtSrcFlow": 1.202867922686202,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            },
            {
                "AvPrecipitation": 8.713893333333337,
                "AvEvapoTrans": 1.0264727503765423,
                "AvGroundWater": 4.027142070919428,
                "AvRunoff": 3.2386926816475,
                "AvStreamFlow": 8.508798272676003,
                "AvPtSrcFlow": 1.2429635201090756,
                "AvTileDrain": 0.0,
                "AvWithdrawal": 0.0
            }
        ],
        "Loads": [
            {
                "Source": "Hay/Pasture",
                "Sediment": 906.4157983340926,
                "TotalN": 14.228528225910749,
                "TotalP": 4.540275448906245
            },
            {
                "Source": "Cropland",
                "Sediment": 339.5240783632521,
                "TotalN": 15.649501742182158,
                "TotalP": 1.799100543191853
            },
            {
                "Source": "Wooded Areas",
                "Sediment": 212.79124106637443,
                "TotalN": 35.51335343479423,
                "TotalP": 2.006691470069048
            },
            {
                "Source": "Wetlands",
                "Sediment": 1.0869708305827872,
                "TotalN": 43.73424842347138,
                "TotalP": 2.302505265375395
            },
            {
                "Source": "Open Land",
                "Sediment": 207.82694777512648,
                "TotalN": 38.85723400030074,
                "TotalP": 0.9186667663928846
            },
            {
                "Source": "Barren Areas",
                "Sediment": 1.385778001625454,
                "TotalN": 3.5256623177820465,
                "TotalP": 0.1184462201334566
            },
            {
                "Source": "Low-Density Mixed",
                "Sediment": 14303.270578379526,
                "TotalN": 389.6824839377943,
                "TotalP": 41.319078536675704
            },
            {
                "Source": "Medium-Density Mixed",
                "Sediment": 147382.60487996537,
                "TotalN": 3980.3396349579652,
                "TotalP": 412.2382289969066
            },
            {
                "Source": "High-Density Mixed",
                "Sediment": 199796.35606070975,
                "TotalN": 5395.869855851091,
                "TotalP": 558.8427212938925
            },
            {
                "Source": "Low-Density Open Space",
                "Sediment": 12350.084036306916,
                "TotalN": 336.469298943642,
                "TotalP": 35.6767418636447
            },
            {
                "Source": "Farm Animals",
                "Sediment": 0,
                "TotalN": 118.88676257230176,
                "TotalP": 30.520851515386198
            },
            {
                "Source": "Stream Bank Erosion",
                "Sediment": 5073942,
                "TotalN": 1340,
                "TotalP": 891
            },
            {
                "Source": "Subsurface Flow",
                "Sediment": 0,
                "TotalN": 280434.9615427279,
                "TotalP": 2146.759264723012
            },
            {
                "Source": "Point Sources",
                "Sediment": 0,
                "TotalN": 9391.891169596127,
                "TotalP": 40582.368633811064
            },
            {
                "Source": "Septic Systems",
                "Sediment": 0,
                "TotalN": 8622.585443999997,
                "TotalP": 0.0
            }
        ],
        "SummaryLoads": [
            {
                "Source": "Total Loads",
                "Unit": "kg",
                "Sediment": 5449443.346369732,
                "TotalN": 310162.19472073123,
                "TotalP": 44710.41120645465
            },
            {
                "Source": "Loading Rates",
                "Unit": "kg/ha",
                "Sediment": 663.692130653497,
                "TotalN": 37.77490557810825,
                "TotalP": 5.445317290209803
            },
            {
                "Source": "Mean Annual Concentration",
                "Unit": "mg/l",
                "Sediment": 84.03247915563007,
                "TotalN": 4.782818446969861,
                "TotalP": 0.6894514648452934
            },
            {
                "Source": "Mean Low-Flow Concentration",
                "Unit": "mg/l",
                "Sediment": 88.37639906385692,
                "TotalN": 5.320213722209787,
                "TotalP": 0.9585043511908502
            }
        ],
        "inputmod_hash": "",
        "watershed_id": null
    },
    "error": "",
    "started": "2022-02-16T20:33:11.918898Z",
    "finished": "2022-02-16T20:33:16.743639Z"
}

Notes

This currently does not have support for validation of the input #3484, or for modifications to the input. Modifications can be made to the JSON directly and submitted, but the modifications + job_uuid support will arrive in the future #3485.

Testing Instructions

  • Start a gwlf-e/prepare job
  • Send its result as input to gwlf-e/run
    • Ensure that works
  • Send the prepare job_uuid to gwlf-e/run
    • Ensure that works, and the results are identical to the step before
  • Inspect the Swagger documentation and offer suggestions for improvements

@rajadain rajadain added the DRWI Funding Source: Delaware River Watershed Initiative label Feb 16, 2022
@rajadain rajadain mentioned this pull request Feb 18, 2022
Merged
4 tasks
@rajadain
Standardize job statuses
9368acd 
We've been using string values for a long time, which have
thankfully not diverged. By wrapping them in constants we
ensure their standard use throughout.
@rajadain
Add custom exceptions for two-step endpoints
1760f10 
For modeling, sometimes an endpoint depends on the output of
a preceding preparatory action. To handle cases when a gwlf-e/run
is given a job that is not ready yet, or has failed, we add
errors. They use the precondition required and precondition failed
HTTP status codes.
@rajadain
Add public GWLF-E Endpoint at gwlf-e/run
85168da 
This takes the output of gwlf-e/prepare and runs it.
If the prepare job is not ready or has failed, returns an
error.

This endpoint is based on the internal start_gwlfe endpoint.
Currently, it does not handle modifications. These will be
added at a later date.
emilyhu0106
emilyhu0106 approved these changes Feb 22, 2022
View reviewed changes
Copy link
Contributor

@emilyhu0106 emilyhu0106 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This endpoint works very well and the description is clear and helpful!

The current description for gwlf-e/prepare is when talking about the next step

The result should be used with the gwlf-e/run endpoint.

Since gwlf-e/run and take both result and job_uuid from gwlf-e/prepare, I think it is helpful to update the description section for gwlf-e/prepare so that users know they can also use the easier-to-paste job_uuid to run the job as well.

@rajadain
Copy link
Member Author

Since gwlf-e/run and take both result and job_uuid from gwlf-e/prepare, I think it is helpful to update the description section for gwlf-e/prepare so that users know they can also use the easier-to-paste job_uuid to run the job as well.

Excellent idea. Will implement this.

@rajadain
Copy link
Member Author

Added in 991e6df.

@rajadain
Re-enable tests
b93cfbc 
These are passing now.
@emilyhu0106
Copy link
Contributor

The new description looks great!

@rajadain
Copy link
Member Author

Thanks. I've been trying to get the CI to pass, but it keeps failing at the same spot, which I'm not able to recreate locally. In the interest of time I'm going to merge this PR. We have #3442 for fixing tests down the line if we need to.

@rajadain rajadain merged commit cf87d7d into develop Feb 23, 2022
@rajadain rajadain deleted the tt/add-public-gwlfe-endpoint branch February 23, 2022 17:49
@rajadain
Copy link
Member Author

Thanks for reviewing!

@rajadain rajadain mentioned this pull request Mar 3, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emilyhu0106 emilyhu0106 emilyhu0106 approved these changes

Assignees

@emilyhu0106 emilyhu0106

Labels
DRWI Funding Source: Delaware River Watershed Initiative
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rajadain @emilyhu0106