Extending Core Types and Descriptors

The Ed-Fi Dashboards application uses a data store aligned with the Ed-Fi Data Standard to populate the metrics and other data displayed. The Ed-Fi Data Standard is extensible, meaning that the standard defines architectural patterns for adding custom data elements to serve specific implementations. The approach to extending the Ed-Fi Dashboards application is harmonized with the way that the Data Standard itself is extended. This section describes the key concepts and processes required to extend the Ed-Fi Dashboards.

For general information on extending the Ed-Fi Data Standard, see the Developers' Guide documentation for the standard. Extensions to the Ed-Fi Data Standard have implications on the ETL processes. The next section will help you understand the necessary steps to take when the schema is extended and new enumerations or descriptors are needed.

Extending or Modifying Enumerations

A common customization to the Ed-Fi Data Standard is a modification to enumerations. This section discusses the steps required to implement a modification to enumerations.

Import Type Data Sheet

The Import Type Data Sheet is a spreadsheet with pre-loaded enumeration values used in the Ed-Fi Data Standard. These values represent the vocabulary that the Ed-Fi technology “understands” out of the box. For referential integrity purposes, the information contained in this file must be loaded prior to loading any other data. By convention, all the enumeration tables in the dashboard are named with the “Type” suffix. Generally, the Import Type Data Sheet spreadsheet contains a tab named for each table, but this naming convention is not mandatory (indeed, there is a character limit when naming spreadsheet tabs in Excel, which may prevent you from adhering to the convention).

An enumeration tab is typically composed of the following elements:

• A Surrogate Key: a unique identifier not derived from business data. This number is created manually and new enumeration values are given the next unused number.
• Code Value: a code or abbreviation that is used to refer to an enumeration value.
• Short Description: a shortened description for the enumeration value.
• Description: The description for the enumeration value.

In some instances, as in the illustrative image above, Short Description is not in the spreadsheet. Not all Type tables contain the ShortDescription column. Additionally, some ETL packages are written to truncate the Description value to 75 characters and insert in the ShortDescription column. If this behavior is not desired, the Import Core Types Loader ETL package (discussed below) can be modified.

Import Core Types Loader

The Ed-Fi Core Types Loader is the SSIS package that extracts information from the Import Ed-Fi Core Types spreadsheet and loads all of its data into the ODS. The package is composed of multiple data flow tasks, organized alphabetically by Type table name.

Inside each data flow task, the basic strategy used to move data from the spreadsheet to the corresponding table is:

1. A SourceConnectionExcel is used in order to connect to the Excel file and extract data from the specified tab. 
   
2. A transformation is used to convert the datatype of the surrogate key to load into the destination Type table. This component is also used to truncate the Description value to 75 characters to load the ShortDescription when necessary.
3. An OLE DB destination is then added to load the data into the respective Type table in the dashboard data store.

Extending Enumeration Types

For every enumeration defined in the Ed-Fi Data Standard and its XSD and ODS expressions, there is a corresponding tab in the Excel file with enumeration values which are loaded into the respective table in the ODS.

To extend the values of an existing enumeration, add the new values to an extension of the enumeration in an extension XSD (see the Extension Framework Guideline documentation for detail), then add the same values in the corresponding tab in the Excel workbook. Lastly, execute the Import Core Type loader package.

Creating New Enumerations

The Extensibility and Configuration section of this documentation discusses the method for creating new enumerations and base types. If it is necessary to have a new base type with corresponding enumerations in the ODS, do the following:

1. Create a new tab in the Excel workbook with the name of the new Type.
2. Enter in the values that have been added in the extension XSD, assigning a unique ID for each enumeration value.
3. Create the new table in the ODS using the same name defined on the XSD.
4. Add a new data flow to the Import Ed-Fi Core Types.dtsx loader to process the new tab in the Excel workbook.

Working with Ed-Fi Descriptors

Ed-Fi Descriptors are conceptually similar to enumerations in that they provide a fixed list of values for a data element or database field. However, the Ed-Fi Descriptor pattern allows a flexible, data-based way to modify the list of enumerations. This more flexible pattern is used in parts of the Ed-Fi Data Standard where per-implementation change or customization is expected, such as a list of courses available at a school, or a set of local discipline incident codes.

Import Descriptor Data Sheet

The Import Descriptor Data Sheet is a spreadsheet containing default Ed-Fi Descriptor values. These descriptor values are already “mapped” to their respective Ed-Fi enumeration values, but can be customized. The information contained in this file must be loaded right after the load of the Type tables for referential integrity purposes. Similar to enumerations Descriptor tables are named with the “Descriptor” suffix by convention. Generally, the Import Descriptor Data Sheet spreadsheet contains a tab named for each table, but this naming convention is not mandatory (indeed, there is a character limit when naming spreadsheet tabs in Excel, which may prevent adherence to the convention). Two spreadsheet tabs define each descriptor: the Descriptor tab and the Type Descriptor tab.

The Descriptor tab is composed of the following elements:

• DescriptorId: A unique identifier in the Descriptor table not derived from business data. This number is created manually and new descriptor values are given the next unused number.
• Namespace: Identifies the namespace of the Descriptor.
• Code Value: A code or abbreviation that is used to refer to a Descriptor value.
• Short Description: A shortened description for the Descriptor value.
• Description: The description for the Descriptor value.
• PriorDescriptorId: The DescriptorId of the Descriptor value that this enumeration is replacing (optional).
• EffectiveBeginDate: The start date that this Descriptor value will be in use.
• EffectiveEndDate: The end date that this Descriptor value will no longer be in use (optional).

The Type Descriptor tab is typically composed of the following elements:

• Type DescriptorId: A unique identifier in a Descriptor type table not derived from business data. This number is created manually and new descriptor values are given the next unused number.
• DescriptorId: A unique identifier that was created in the Descriptor tab.
• Type Id: A unique identifier of the corresponding enumeration value that “maps” to this descriptor value (this can be seen in the Import Ed-Fi Core Types.Data.xls).

Import Descriptor Loader

The Ed-Fi Descriptor Loader is the SSIS package that extracts information from the Import Ed-Fi Descriptor spreadsheet and loads the data into the ODS. It is composed of multiple data flow tasks, one that loads data into the edfi.Descriptor table, and the rest to load each type descriptor table (e.g., edfi.AcademicSubjectDescriptor).

Inside a data flow task, the basic structure followed to move data from the spreadsheet to the corresponding table is as follows: First, a SourceConnectionExcel variable is used in order to connect to the Excel file and extract data from the specified tab. 

Next, a transformation is applied to convert the values of the surrogate key to the necessary data type to load into the destination table. 

Finally, an OLE DB destination is then added to load the data into the respective type Descriptor table.

Extending Descriptors

For every Ed-Fi Descriptor in the data model, values are defined in the Descriptor tab and a corresponding Type Descriptor tab in the Excel workbook (e.g., the AcademicSubjectDescriptor tab). An implementation adding a new Descriptor should first define the use based on business requirements, and understand what existing descriptors are present in the unmodified Ed-Fi Data Standard. This will determine one of three types of implementations:

• Descriptor with no Map value. This type of implementation does not use an enumeration (or Type table), and only requires a new type Descriptor table.

• Descriptor using an existing Map value. This type of implementation uses an existing enumeration (and Type table), and only requires a new type Descriptor table.

• Descriptor using a new Map value. This type of implementation requires the creation of a new enumeration (and Type table), as well as the new type Descriptor table.

Further Reading

The next section, Walkthrough Example: Adding Custom Grade Level Descriptors, applies the concepts discussed above.