General Synchronization algorithm overview Synchronization usually done with dates or with an artificial change version Ed-Fi is working with an artificial change version, in this case a database level sequence which will ensure each create/update has a unique long value Using long means we have 9,223,372,036,854,775,807 change versions, so not likely to run out Always an option to reset the sequence (and change versions), but would force all end users to full sync Syncs always involve setting a known start and end range Never sync with an open max, since data can be volatile and new records might come in while syncing Open min is ok, because 0 represents a static edge to work against. Always have to work in reverse dependency order if you're pulling data into a system with relational integrity. Basic syncing would involve getting the max change version, then pulling all data in the system up to that change version. Delta syncs after that would use the previously retrieved max change version as the new min version, and retrieve a new max change version to use. Problem is syncing errors can occur, due to volatile underlying data (e.g., you retrieve students, someone submits a new student and student section association while you're grabbing assessment data, you get a failure when you go to pull student section associations) To work around this, recommend always syncing two change periods, and doing a delta sync immediately after your initial sync. This will mitigate the number of resources that error out due to volatile data by adding some amount of reprocessing. For example, assuming a daily sync: Initial Sync: minChangeVersion = NULL, maxChangeVersion = 100 Delta Sync #1 (immediately after initial sync): minChangeVersion = NULL, maxChangeVersion = 110 Delta Sync #2 (1 day later): minChangeVersion = 100, maxChangeVersion = 230 Delta Sync #3 (2 days later): MinChangeVersion = 110, MaxChangeVersion = 370 Delta Sync #4 (3 days later): MinChangeVersion = 230, MaxChangeVersion = 435 Full Sync Example No impact on existing url structure, so normal gets still function as expected EXAMPLE: Reading Data -> GetAllPrograms For full sync, we need to find out what the current max change version in the system is EXAMPLE: Reading Data -> GetAvailableInitialSync Show Tests and resetting the three variables representing change versions for the two change periods we care about Then get data for each resource based on that max change version EXAMPLE: Reading Data -> GetProgramByChangeVersionInitialSync Perform for each resource Then perform delta sync EXAMPLE: Modifying Data -> GetAvailableDeltaSync Show Tests and setting three variables based on moving the change version back so it represents the two change periods EXAMPLE: Modifying Data -> GetProgramByChangeVersionDeltaSync No data returned since hopefully nobody is updating programs in my sandbox Add/Update Example Show variables and how the change period adjusts as we run these. In between syncs, data comes in. New program is added. EXAMPLE: Modifying Data -> PostProgram EXAMPLE: Modifying Data -> GetProgramByKey Show the new record having a higher change version EXAMPLE: Modifying Data -> GetAvailableDeltaSync EXAMPLE: Modifying Data -> GetProgramsByChangeVersionDeltaSync Returns the one new record Also supports programs being updated between syncs EXAMPLE: Modifying Data -> PutProgramWithUpdatedProgramId EXAMPLE: Modifying Data -> GetProgramByKey Show the modified record changing the change version EXAMPLE: Modifying Data -> GetAvailableDeltaSync EXAMPLE: Modifying Data -> GetProgramsByChangeVersionDeltaSync Returns the one updated record