diff --git a/README.md b/README.md index 0c2f4c8..74d1510 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # DataHubAPIDemo Demonstration scripts and notebooks showing how to use the Data Hub API for data submissions, reporting, and other tasks. -**Note:** All of these scripts will require a Data Hub API key in order to use. Instructions for obtaining an API token can be found in the [data submission documentation](https://datacommons.cancer.gov/data-submission-instructions) +**Note:** All of these scripts will require a Data Hub API key in order to use. Instructions for obtaining an API token can be found in the [data submission documentation](https://datacommons.cancer.gov/data-submission-instructions). For security reasons, these tokens should be stored as environment variables on your system. The scripts expect production and stage API keys to be stored in the environment variables PRODAPI and STAGEAPI, respectively. ## DataHubAPIDemo.ipynb This Jupyter notebooks walks through a basic example of how to do a CRDC submission using the Data Hub APIs. @@ -23,7 +23,10 @@ This notebook covers several queries that can provide more detailed information - Retrieving a populated configuration file for use in uploading data files with the CLI Upload Tool ## SubmissionReportDashboard.py -This is a Python Dash application that uses the APIs to create a personal dashboard of your submissions. +This is a Python Dash application that uses the APIs to create a personal dashboard of your submissions. To use this script, run the script (# python3 SubmissionReportDashboard.py), then launch a browser and navigate to http://localhost:8050. + +**Required Python Libraries** +dash, dash_bootstrap_components, plotly, requests, pandas, datetime, pytz ## ShinyDashboard.py @@ -34,6 +37,36 @@ Simialr to the SubmissionReportDashboard only uses Python Shiny instead of Dash. Submissions that are inactive for extended periods of time start generating warning emails and after 180 days get deleted. The remedy to this situation is to log into the Submission Portal and look at the submission. However, this gets burdensome if there are a large number of submissions to check. This script (also in notebook form) will query for all the submissions that are either New or In Progress and will request information from each of them. This re-sets the inactvitiy timer. ## WarningAggregator.ipynb and WarningAggregator.py +Currently, the Submission Portal does not have aggreation for the Warnings that are generated by data updates. This can make it difficult to see if the updates that are about to be applied are correct. These programs will aggregate all the warnings and display them in a paired manner so that it is easier to see what changes are about to be applied. + +Both scripts take the following configurations. In the notebook you will find these in a marked cell, edit that cell to need. For the script, the configurations are provided in a YAML file (see *warning_configs.yml* for an example) + +**subid**: A list of the submission IDs to be checked. Submission IDs can be obtained from the Submission Portal +**severity**: Should be set to 'All' +**nodelist**: A list of the nodes that should be checked for Warnings +**outputdirectory**: A local directory where output can be written +**tier**: The tier to use, shoudl be either 'stage' or 'prod' + +**Script runtime options** + - -c/--configfile: The YAML configuration file + - -v/--verbose: The level of verbosity. Add more v's to be more verbose. + + +## SubmissionReset.py +This script resets the inactivity timer for all of your **New** or **In Progress submissions** to the current date. + +**Script runtime options** +- -t/--tier: The tier to check. Should be either 'prod' or 'stage' +- -v/--verbose: The level of verbosity. Add more v's to be more verbose. + +## SubmissionResetGUI.py +This is a graphical version (Python Dash) of the *SubmissionReset.py* script. Select a tier from the drop-down and a table of your current New and In Progress submissions will be generated. Select the submissions you wish to reset the inactivity timer on and then click on the *Reset Time on Selected Submissions* button below the table. Each submission selected will be reset to the current date. + +To use, run the script ($ python3 SubmissionResetGUI.py) and then bring up a browser and navigate to http://localhost:8050. + +**Required Python Libraries** +dash, dash_bootstrap_components, requests, pandas, datetime, pytz + When updating a submission that has previously been through DataHub, it's possible to get a great number of warnings that data is going to be changed. Unfortunately, the current Submission Portal interface doesn't have a way to aggregate and display these warnings which can make it difficult and tedious to check. This script and notebook will aggregate all the warnings in a submission and display alternating old and new lines in a table(notebook) or output a csv file (script). ## DeleTron.py @@ -50,4 +83,4 @@ DeleTron requires a yaml file with the following parameters. It is recommended - **submissionid**: The UUID for the submission you are editing. This can be copied from the upper left of the submission view in the GUI. - **node**: The node you will be deleting informaiton from. For example *file*, *diagnosis*, or *participant* -There is additional required information in the **mdffiles** section that should not be edited. If you create your own yaml configuration file, make sure this section is copied over and not edited. \ No newline at end of file +There is additional required information in the **mdffiles** section that should not be edited. If you create your own yaml configuration file, make sure this section is copied over and not edited. diff --git a/SubmissionResetGUI.py b/SubmissionResetGUI.py new file mode 100644 index 0000000..ebcfe47 --- /dev/null +++ b/SubmissionResetGUI.py @@ -0,0 +1,322 @@ +from dash import html, dcc, dash_table, Output, Input, State +import dash +import dash_bootstrap_components as dbc +import requests +import pandas as pd +import io +from datetime import datetime, timezone +from pytz import timezone as tz +import os + +####################################### +# # +# App Definition # +# # +####################################### + +external_stylesheets = [ + { "href": "https://fonts.googleapis.com/css2?" + "family=Lato:wght@400;700&display=swap", + "rel": "stylesheet", + }, + dbc.themes.BOOTSTRAP +] + +app = dash.Dash( + __name__, + external_stylesheets=external_stylesheets, + suppress_callback_exceptions=True, + prevent_initial_callbacks=True, + update_title="Updating..." +) +app.title ="DH Dashboard" + +############################################ +# # +# Styles # +# # +############################################ + +CONTENT_STYLE = { + "margin-left": "18rem", + "margin-right": "12rem", + "padding": "2rem 1 rem" +} + + + + +####################################### +# # +# Subroutines # +# # +####################################### + + +def dhAPICreds(tier): + url = None + token = None + if tier == 'production': + url = 'https://hub.datacommons.cancer.gov/api/graphql' + token = os.getenv('PRODAPI') + elif tier == 'stage': + url = 'https://hub-stage.datacommons.cancer.gov/api/graphql' + token = os.getenv('STAGEAPI') + elif tier == 'qa': + url = 'https://hub-qa.datacommons.cancer.gov/api/graphql' + token = os.getenv('QAAPI') + elif tier == 'qa2': + url = 'https://hub-qa2.datacommons.cancer.gov/api/graphql' + token = os.getenv('QA2API') + elif tier == 'dev': + url = 'https://hub-dev.datacommons.cancer.gov/api/graphql' + token = os.getenv('DEVAPI') + elif tier == 'dev2': + url = 'https://hub-dev2.datacommons.cancer.gov/api/graphql' + token = os.getenv('DEV2API') + elif tier == 'localtest': + url = 'https://this.is.a.test/url/graphql' + token = os.getenv('LOCALTESTAPI') + return {'url': url, 'token': token} + + +def dhApiQuery(url, apitoken, query, variables=None): + headers = {"Authorization": f"Bearer {apitoken}"} + try: + if variables is None: + result = requests.post(url=url, headers=headers, json={"query": query}) + else: + result = requests.post(url=url, headers=headers, json={"query": query, "variables": variables}) + if result.status_code == 200: + return result.json() + else: + return (f"Status Code: {result.status_code}\n{result.content}") + except requests.exceptions.HTTPError as e: + return (f"HTTPError: {e}") + + +def getSubmissionData(tier): + tier = tier.lower() + creds = dhAPICreds(tier) + list_sub_query = """ + query ListSubmissions($status: [String]!){ + listSubmissions(status: $status){ + submissions{ + _id + name + submitterID + submitterName + studyAbbreviation + studyID + dbGaPID + createdAt + updatedAt + metadataValidationStatus + fileValidationStatus + status + } + } + } """ + + variables = {"status": ["New", "In Progress"]} + results = dhApiQuery(creds['url'], creds['token'], list_sub_query, variables) + return results + + + +def elapsedTime(submission_df): + days = [] + for index, row in submission_df.iterrows(): + temp = row['updatedAt'].split('T') + update = datetime.strptime(temp[0], '%Y-%m-%d') + update = update.replace(tzinfo=tz('UTC')) + now = datetime.now(timezone.utc) + diff = (now - update).days + days.append(diff) + submission_df.insert(8,'inactiveDays',days,True) + return submission_df + + +def updateSubmissionClock(subid, tier): + getSubmissionQuery = """ + query GetSubmissions( + $id: ID! + ){ + getSubmission(_id:$id){ + _id + name + dataCommons + } + } + + """ + vars = {"id": subid} + creds = dhAPICreds(tier) + #print(f"Vars: {vars}\n Creds: {creds}\n") + updatejson = dhApiQuery(creds['url'], creds['token'], getSubmissionQuery, vars) + #updatejson = {} + return updatejson + + + + +############################################ +# # +# Components # +# # +############################################ + +pageheader = html.Div([ + html.Div( + html.H1('Submission Update Tool'), + style={'textAlign':'center'} + ), + html.Div( + children='A tool to reset the submission clock', + style={'textAlign':'center'} + ) +]) + +dropdown = html.Div([ + html.Div( + className='tierdropdown', + children=[ + dcc.Dropdown( + id='tierselector', + options=[ + {'label': 'Production', 'value': 'production'}, + {'label': 'Stage', 'value': 'stage'}, + {'label': 'QA', 'value': 'qa'}, + {'label': 'QA2', 'value': 'qa2'}, + {'label': 'Dev', 'value': 'dev'}, + {'label': 'Dev2', 'value': 'dev2'} + ], + placeholder='Select a tier', + multi=False, + style={'backgroundcolor':'1E1E1E'}, + ), + ] + ) +]) + + +errormessages = html.Div(id='printhere', style={'whiteSpace':'pre-line'}) + +updateButton = html.Button('Reset Time on Selected Submissions', id='updatethis', n_clicks=0) + + +tableheader = html.Div([ + html.H2("Your current Submissions", style={'textAlign':'center'}), +], + style=CONTENT_STYLE) + +submissioncontent = html.Div( + [ + html.Div(dbc.Spinner(html.Div(id="submissioncontentspinner"), color="primary")), + html.Div(id="submissioncontent") + ] +) + +#################################### +# # +# Layouts # +# # +#################################### + +app.layout = html.Div([ + pageheader, + html.Div([dropdown]), + html.Hr(), + html.Div([tableheader]), + html.Hr(), + html.Div([submissioncontent]), + html.Div([updateButton]), + html.Div([errormessages]), + dcc.Store(id='substore') +]) + + +#################################### +# # +# Callbacks # +# # +#################################### + +################## Datastore callbacks ###############3 +@app.callback( + Output('substore', 'data', allow_duplicate=True), + Input('tierselector', 'value') +) +def updateSubstore(tierselector): + subjson = getSubmissionData(tierselector) + sub_df = pd.DataFrame(subjson['data']['listSubmissions']['submissions']) + sub_df = elapsedTime(sub_df) + + return sub_df.reset_index().to_json(orient='split') +################### Table callbacks ################# + +@app.callback( + Output('submissioncontent', 'children'), + Input('substore', 'modified_timestamp'), + State('substore', 'data') +) +def populateSubmissionsTable2(modified_timestamp, substore): + sub_df = pd.read_json(io.StringIO(substore), orient='split') + return dash_table.DataTable( + id = 'subtable', + data=sub_df.to_dict('records'), + columns=[{"name": e, "id": e} for e in (sub_df.columns)], + row_selectable="multi", + sort_action='native', + sort_mode='multi', + style_table={'overflowX':'auto'}, + style_cell={'overflow':'hidden', 'textOverflow':'ellipsis', 'maxWidth':10, 'textAlign':'center'}, + style_data={'color':'black', 'backgroundColor':'white'}, + style_data_conditional=[{'if':{'row_index':'odd'}, 'backgroundColor': 'rgb(220,220,220)'}], + style_header={'backgroundColor': 'rgb(210,210,210)', 'color':'black', 'fontWeight':'bold', 'textAlign':'center'}, + tooltip_data=[ + { + column:{'value': str(value), 'type':'markdown'} + for column, value in row.items() + } for row in sub_df.to_dict('records') + ], + tooltip_duration=None, + export_format="csv" + ) +######################## button callback ********************** +# +# https://community.plotly.com/t/how-to-get-the-data-of-the-selected-rows-of-dash-table-experiments/8439 +# +@app.callback( + Output('substore', 'data', allow_duplicate=True), + Input('updatethis', 'n_clicks'), + State('subtable', 'selected_rows'), + State('substore', 'data'), + State('tierselector', 'value') +) + +def updateSubmissions(n_clicks,selected_rows, substore, tierselector): + if n_clicks >= 0: + sub_df = pd.read_json(io.StringIO(substore), orient='split') + selected_df = sub_df.iloc[selected_rows] + collected_res = [] + for index, row in selected_df.iterrows(): + print(row['_id']) + res = updateSubmissionClock(row['_id'], tierselector) + collected_res.append({row['name']: res}) + + subjson = getSubmissionData(tierselector) + sub_df = pd.DataFrame(subjson['data']['listSubmissions']['submissions']) + sub_df = elapsedTime(sub_df) + + return sub_df.reset_index().to_json(orient='split') + +#################################### +# # +# Run Program # +# # +#################################### + + +if __name__ == "__main__": + app.run(port=8050, debug=True) \ No newline at end of file