Data Import Hands On Lab

Overview

This lab provides an end-to-end example of using Data Import highlighting its major functionality:

• Reuse a template from the Template Sharing Service as a starting point
• Review data mapping obtained from template
• Setup an agent for CSV file importing
• Import a sample state assessment into an Ed-Fi ODS / API
• Review logs after import to verify and QA

Data Import Lab

This section contains a simple walkthrough which can be used to verify installation and perform a simple end-to-end import using an Ed-Fi ODS / API v3.2 and the Grand Bend sample data set.

The high-level steps are:

Step 1. Connect to VM

Connect to your lab VM using: VM Connection Info.

Step 2. Open Data Import in the Web Browser

Once logged into your VM, click on the "Ed-Fi Data Import" icon on the desktop which will launch the local instance of Data Import. 

Log in using the details below:

If everything is configured correct, you should see a Data Browser listing of Schools available.

Step 3.  Review Use Case for Importing Sample State Assessments

The desktop of your lab VM is a file named "DI-TEST-2019-GrandBend.csv".  Double click on this file to open and select Notepad when selected.

Observe the CSV file with 10 rows of sample state assessment scores for Grand Bend.  View the columns to have an understanding of the data we'll be importing.

Step 4. Obtain sample from Template Sharing Service

Data Import has the ability to import and export templates (data maps, lookups and bootstraps) and share those templates to other Ed-Fi adopters in the community.  We'll use this feature to download samples for this lab.

Click on Admin → Template Sharing to access this.

This screen allows for publishing and consuming of Data Import templates as shared by other adopters.  Notice the Grand Bend 2019 templates, select the one for Ed-Fi ODS / API Version "3.2.0" and click View.

This is a sample template.  View the information below.  The template preview provides a view of what will be imported.  It is important to note, Data Import is intended to share only metadata for import operations and not actual results or other real data.  Click on Import Template.

Step 5. Review Lookups

Click on menu option Lookups.  Observe the "grade-level" lookups that have been imported from the template sharing service.

Step 6. Review Bootstraps

Click on Admin → Bootstrap Data.  Observe that four bootstraps have been imported for template sharing.  Click on the first two and view its data below.

Review "Grand Bend 2019 State ELA Assessment".  Click Back or Save.

Review "Grand Bend 2019 State Objective Assessments".  Click Back or Save.

Step 7. Review Data Map

Click on menu option Maps.  Review the "Grand Bend 2019 Sample...." that was imported by the template sharing service.  Click Edit to review.

To preview the data source, click "Choose File", the "DI-TEST-2019-GrandBend.csv" from the Desktop, then click Upload.

The data preview will appear.  Browse and review various aspects of this data map.  Notice that it uses column, static and lookups within this map.

Step 8. Add an Agent

Agents associate a file transfer with the maps defined for this file.

Click menu option Agents.  Click Add Agents.

Enter options as described below.  

Click Save Agents when done.

Step 9. Upload file to Agent

Once you have created the agent, you must upload the CSV file downloaded earlier ("DI-TEST-2019-GrandBend.csv") because this is a manual agent. On the main Agents page, click the "Upload" link on the agent you just created. Choose the CSV file and click Upload. Back on the main Agents page, the "Files" column for that agent should now be 1. Now, we are ready to load to the ODS.

Step 10.  Run the Agent

Execute Data Import's Transform Load command line task to import and transform CSV data to the StudentAssessment endpoint. This can be done by double clicking the icon on the desktop titled "DataImport - Shortcut" which will run the DataImport.Server.TransformLoad.exe from the TransformLoad folder.

Step 11. View Logs

Data Import provides logging facilities for files to process, ingestion log details, and a general application log. Select the menu item Logs to view this section and details about the previous import job.

Review job status which should now read "Loaded"

Click on the Ingestion tab.  Review the details below.  Click the "+" to open extended information about the API payload for that row.

Support & Reporting Issues

If you need support while using Data Import, you can log an issue in the Ed-Fi Tracker.