DMM - Importing

The Data Migration Manager (DMM) provides a mechanism to bulk import and edit historical data, such as data loaded from Excel.

The DMM offers bulk validation and import of Locations, Wells, Field Chemistry, Historical Chemistry, Bore Logging and Geology data, as shown below.  It also facilitates data transfer from other ESdat systems using the ESdat Data Interchange Format.

Loading Data from Excel

Data can be entered directly in the online worksheet, or blank Excel Import Templates can be downloaded, populated offline, then used to populate the worksheet in ESdat.

If you generate an Excel Import Template, once populated it will look like below.  Columns with an asterix (*) in the header are required fields and must be populated with data.  Data that is identified in Excel for importing will be visible by alternating blue and white rows as shown below. 

Populated Excel Import Templates can be loaded, or data can be diretly pasted into the online worksheet.  Data in the online worksheet will look as below.

Loading Data from an ESdat Data Interchange Format

All ESdat systems, including legacy ESdat Desktop systems, are able create an ESdat Data Interchange File which can be used to move data between ESdat systems.

A Data Interchange File can be loaded using the Data Interchange button, shown below.

The data will load in a series of sheets, ready for Validation and Import.

Validating and Importing Data

Drop-Downs are available to validate the data.

To import data the user will need to specify a Project for which they have Add Permissions.   If the user is looking to update/overwrite existing data they will need to specifiy a project for which they have Edit permission. 

	Add	Add new records. Rows will only be processed if the record does not already exist.
	Overwrite	Overwrite existing records. Rows will only be processed if the record already exists.   Any existing data for the row will be overwritten, including with null values (meaning a value in the database will be overridden, potentially by an empty value).
	Update	Update existing records. Rows will only be processed if the record already exists.   Existing data for the row will be updated where a cell has a value.  Empty cells will be ignored  (meaning any existing data in the corresponding field already in ESdat will remain unchanged).
	Add/Update	Add/Update existing records. Rows will be processed as an Update if the record already exists, otherwise they will be Added.

Prior to importing the data into ESdat, you may choose to validate the data in your worksheet.  

The validation checks for the following:

• Required fields must be populated with data.
• Correct Data Types, for example the numerical field of x_coord must only contain numbers.
• Field lengths are enforced
• If the record (such as a Location Code) is already present in the database then it can not be added (which would create a duplicate), an update or overwrite needs to be used.
• Invalid reference data will not be imported. For example if an unknown monitoring zone, Site or Project is referenced on a Location it will be reported as an error.
• The logged in user must have add permissions for the projects they are importing data against, and edit permissions for the project if they are updating.  
• If a new site and project association is made in the data upload it will be highlighted for confirmation before being created in the system. 

If any of these checks fail a summary the import errors are displayed to the user.  The user can filter for specific errors, and a tooltip provides feedback on the error in each row as in the example below:

Successfully validated data will be imported, while rows with errors will be flagged and not imported.  Once you have resolved any Validation issues, click the Import button to import the corrected data into ESdat.  

Chemistry Considerations

If you are loading Chemistry data from Excel or an ESdat Data Interchange File you will need to ensure your data is using the standard ESdat ChemCode Reference List.  For more information on loading Chemistry see the specific "Importing Historical Chemistry" document.

Saving Imports and Metadata

Every data import is automatically saved. It is also possible to add metadata to describe an import.

All saved imports and can be reopened.  When it is no longer needed a saved import can be deleted.