# s3ds

This is an early toolbox designed for creating and validating data in the s3ds* prototype "standard"** . This should be considered a working prototype for packaging subsurface data into a structure that mimics familiar and recognized conventions of the GeMS standard for geologic maps.

***This schema and its accompanying validation do not constitute a formal set of requirements. For the time being they are merely here to facilitate community engagement and discussion toward an eventual standard.***

Early reviewer comments were keen to draw distinction between data elements required for 3D geologic models and those used in describing geologic maps. For this reason we describe here a structure that can stand-alone from GeMS but should be quite familiar to those using and preparing geologic maps in that standard.

 You can read more about this document here: [![View the draft document](https://img.shields.io/badge/-View_the_draft_document-blue)](markdown_docs/DraftDocumentation.md) 

<div style="background: #ffffff86; color: #000">
<figure>
<img
src="media/cartoon_example.png" />
<figcaption><p>Figure 1. Example of the types of content intended to be captured in the s3ds prototype standard.
</p></figcaption>
</figure>
</div>

More generally, this is a toolset for creating and validating relational geospatial data stored in `.gdb` or `.gpkg` formats. While we have developed this codebase for exploring the proposed s3ds schema, we have tried to build the codebase to be nimble to changes in the structure that will come from peer review.  As a result, the code is built to just respond to a schema definition provided as a JSON file that defines tables, fields, datatypes, uniqueness, nullability, and foreign key constraints. The tools for building and validating that database are responsive to changes in that schema definition (with the ability to add some extra data creation and validation options in as necessary).

*For the time being, we refer to this as S3DS (pronounced; “seds” for “Subsurface 3D Schema”). Alternate names are enthusiastically welcomed, we just wanted a placeholder name to distinguish this content from GeMS.

**We are hesitant to call this a "standard" prematurely. We are merely exploring options for a data structure that could allow consistent storage and repositing of 3D geologic models.

## Description of the tool and status
This project is in early development. There are basic 'build' and 'validate' routines available as command line tools (`build_database.py` and `validate_database.py`).

There are equivalent functions available as GUI toolbox tools in ArcPro, instructions for the ArcPro toolbox are here:  [![View ArcProTools](https://img.shields.io/badge/-View_instructions_for_ArcPro-blue)](markdown_docs/WorkingInArcPro.md) 

There are also equivalent functions available as GUI toolbox tools in QGIS, instructions for the ArcPro toolbox are here:  [![View ArcProTools](https://img.shields.io/badge/-View_instructions_for_QGIS-green)](markdown_docs/WorkingInQGIS.md)

The following future additions are envisioned as possible additions to this suite, though some may be beyond the scope of what is needed and add too much complexity:

1) A tool to build FGDC metadata and USGS data dictionaries for a supplied dataset
2) An iteration that can create an SQL/postgis script for reproducing a database structure
3) A tool that can procedurally combine many datasets into one, mainintaing referential integrity and adding some kind of parent-record/citation indicator.

## Description of the data format

The data format is designed to mirror that of GeMS, with specific changes that accomodate the nuances of 3D geologic models. Please consult the document for more on these decisions: [![View the draft document](https://img.shields.io/badge/-View_the_draft_document-orange)](markdown_docs/DraftDocumentation.md) 

The current iteration of the format is shown in the Entity Relationship Diagram in Figure 2, however this toolbox was designed to be nimble and somewhat format agnostic. Rather than a specified format, the code reads the format defined in [tables_defs.json](src/table_defs.json). It *should* respond to a variety of possible relational geospatial formats. For more about this please see this more detailed description of how the schema is specified for this tool:  [![View the schema definition specs](https://img.shields.io/badge/-Defining_data_schemas-red)](markdown_docs/SchemaDef.md) 

<div style="background: #ffffff86; color: #000">
<figure>
<img
src="media/erd.png"/>
<figcaption><p>Figure 2. Current iteration of the ERD for the prototype s3ds standard. NOTE! This figure is out-of-date for the borehole tables in particular. There are now two types of borehole data captured in two seperate paired table structures; one for vertical and one for non-vertical wells. While the general structure is similar please don't rely on this table for accurate documentation. See the discussion below and the files that define the tools themselves. We will work to update this figure as we converge on a structure.</p></figcaption>
</figure>
</div>

### [`build_database.py`](src\build_database.py)
This tool takes four arguments:

1. **database path** the path to a geopackge or geodatabase (including the extension `.gpkg` or `.gdb`) that will be created, or one that exists and should be extended.
2. **Tables** a semi-colon seperated list of desired tables.  The code will determine required tables based on the supplied tables of interest. For example, if only contour lines are being included just suppling `modelelementvaluelines` will ensure the requried `datasources`, `modelelements`, `measurementunits`, and `elementtypedict` are included.
3. **EPSG** --Please specify an EPSG code for the coordinate system you would like to use (e.g., '4326' for WGS84, '32615' for WGS84 UTM Zone 15N, and so on) 
4. **Enforce relational integrity?** -- If true, this will make all `<tablename>_id` values auto-incrementing (so that a user doesn't need to populate them) in geodatabases (they are always autoincrementing in geopackages). This will also ensure that records can not be created in the database if they violate relational integrity. For example, you can not create a record that references a 'datasource' that doesn't exist or doesn't reference a datasource at all. This is accomplished with relationships in geopackages, but uses attribute rules in the geodatabases (which require global ids). 

The function can be called from the command line as follows:

    python <yourinstallpath>\S3DS\src\build_database.py <desiredfolderpath>\my_s3ds_geopackage.gpkg modelelementvaluelines;modelelementsurfaces

If the function is called on a database that already exists it will add any additional required tables to that database.

- [ ] TODO: Update this to be able to work as a module or a command line tool. Currently imports and directory structure are setup to emphasize the later. 

### [`validate_database.py`]((src\validate_database.py))

This tool takes two arguments:

1. **database path** the path to a geopackge or geodatabase (including the extension `.gpkg` or `.gdb`)
2. **output file** (optional) - the path to a text file that will log any validation errors or code exceptions encountered during evaluation. If a text file isn't specified this will just be printed to the console. In the future, we will update this with better formatting/logging.

The function can be called from the command line as follows:

    python <yourinstallpath>\S3DS\src\validate_database.py <basefolder>\my_s3ds_geopackage.gdb <basefolder>\myvalidation.txt

[] TODO: Update this to be able to work as a module or a command line tool. Currently imports and directory structure are setup to emphasize the later. 


### Clone

This method requires you to have git installed on your computer. If you know you do, you can skip down two steps and use that instance of it. If not, you can use the instance that was installed in the default conda environment when ArcGIS Pro was installed (if you are installing for ArcPro).

* Click on 'Start' in the lower-left of your desktop.
* From the list of programs, select ArcGIS and then Python Command Prompt.
* At the prompt, type cd and the path of the location where you would like the toolbox to be.
* Paste in:
`git clone https://code.usgs.gov/atan/dem_getter.git`
* change directory to geologic-contact-fitter-main

Now, whenever you get notice of a new release or if it's just been a while and you want to make sure you are using the most up-to-date version, go back to this folder and run `git pull`.

## Requirements

If you are building a virtual environment to execute this code, the following are dependencies:

* gdal
* ogr
* osr
* sqlite3

To build geodatabases arcpy must also be available. All of these packages are present in the base python environment of ArcPro.


Disclaimer
=======
[![Disclaimer](https://img.shields.io/badge/-Go_to_disclaimer-red)](markdown_docs/Disclaimer_provisional.md) 

License
=======

[![License](https://img.shields.io/badge/-Go_to_license-red)](markdown_docs/License.md) 


## Authors and acknowledgment
This prototype standard is the product of collaboration between individuals in the USGS (including within the FEDMAP and NGMDB components of the National Cooperative Geologic Mapping Program) and from state geologic surveys. For specific questions about this codebase or repository please contact Sam Johnstone: sjohnstone@usgs.gov 


