Migrating digital records

Migrating records

Line of business systems may have a shorter life than the records they contain. Technology changes rapidly and systems regularly improve and are updated. When systems are superseded but the record is still required for business or operational purposes, the records must be transferred to the new system in a process called migration.

Migration processes create a new version of a record, while the old version (source record) of the record is kept for the purposes of quality assurance. After the migration process is completed, source records can often be disposed of. The General authority for source records that have been migrated (GA48) provides the permission for the disposal of source records. The GA48 also includes a number of requirements that must be meet in order to destroy source records and keep the migration process safe and reliable.

What needs to happen before destroying source records?

Before destroying any source records public offices must:

1. Manage the migration process:  Plan and document to ensure the process is organised.
2. Meet requirements: The conditions of disposal of General authority for source records that have been migrated (GA48) must be met before any destruction is undertaken.
3. Test before and after migration: Check that the new records are accurate, complete, easy to access, and usable.
4. Keep source records for an appropriate time: Keep the source records long enough to confirm that the migration was successful.  Determination of the specific retention period must be based on an organisational risk assessment.

Following these steps ensures the new records are trustworthy and can be used properly.

1. Planning for migration

It’s important to carefully plan a migration process to ensure that the records resulting from the migration process are authentic, reliable, complete, accessible, and usable. Additionally, careful planning of the migration will assist in meeting the requirements in GA48 and enabling the public office to destroy the source records.

To understand what is to be migrated, the plan should include:

• understandings of user needs, system usage and process
• identify the information that users input in existing fields and whether the uses of data fields are consistent (i.e. a data mapping)
• identify the data to be migrated and any external data sources that may also need to be migrated
• check the quality of data and clean if necessary
• identify if there are records due for destruction, no longer required for business purposes, which could be disposed of prior to the migration
• identify essential relationships between records and their hardware/software needs
• ensure metadata that supports records’ integrity is preserved and retained, including structural and relational metadata.

Develop an appropriate migration plan

The migration plan should outline:

• data structure, volume, and quality
• migration strategy, including metadata mapping and format transformations
• testing and validation methods (before and after migration)
• risks, mitigation strategies, and recovery plans
• actions on source records and planning for decommissioning
• roles and responsibilities for migration tasks.

The migration plan is a key document, and any project changes should be reflected in it.

2. Maintaining authentic, reliable, complete, and useable records

Successful migration operations will produce authentic, reliable, complete, and useable records.

It is a condition for the destruction of source records that have been migrated that pre and post migration testing proves that authentic, reliable, complete, and useable records can and have been migrated.

An unsuccessful migration can result in:

• loss of information – where specific metadata, context links, or even data quality is lost
• failure to process or display the information – migrating between systems can result in information not displaying correctly due to structural differences between the systems.
• lack of awareness of change – if business procedures and employees are not updated to the changes of migrated records, locating the records can become difficult.

3. Migration

Pre-migration testing

Pre-migration testing helps ensure the migration plan works as expected. It involves testing the migration process on a small sample of records to check for errors. Relevant personnel should verify the records' completeness and accessibility.

Any issues found must lead to a revised migration strategy, which should then undergo further testing. Document the results of testing, which will inform the final migration plan. Once pre-migration testing is complete, the results and finalised plan should be validated as part of the migration project governance.

Post-migration testing

Post-migration testing checks that:

• all identified records and metadata have been migrated successfully
• necessary functionality and essential characteristics are preserved
• users are satisfied with the migrated records.

Testing can be done through sampling, with the sample size proportional to the business risk. Document the results and validate them according to the project governance arrangements.

Incidental quality assurance

Public offices should also consider the extent of ‘incidental’ quality assurance when determining an appropriate retention period for source records.

If data is migrated and then immediately and frequently used in business as usual processes, anomalies and unforeseen issues with the migrated data are more likely to be found, and found within a shorter period of time. In these scenarios, and subject to the findings of the risk assessment, it may be appropriate to retain source records for a short period of time.

Understanding user behaviour and requirements is critical to successful migration projects. If the business is not involved in migration projects, key fields or functionality may not be migrated. In these scenarios, source records should be retained for a longer period so that the business has the opportunity to identify any anomalies or issues.

The migration of data between organisations following administrative change is a scenario where incidental quality assurance is likely to be delayed as users in the receiving organisation take time to use and understand the data, and identify any issues requiring rectification.

The migration of data between organisations is a high risk scenario. It is critical that migration processes are supported by a plan with agreed upon timeframes for quality assurance testing and retention of source records, and agreed upon sign-off arrangements before the destruction of source records by the transferring organisation can proceed.

See Managing records in administrative change for further advice.

Documenting migration

The entire migration process must be documented, as this supports the authenticity and trustworthiness of the migrated records.

Migration should be undertaken in accordance with best practice project management methodologies.

Comprehensive records of the project should document:

• the factors triggering the migration project
• the records being migrated and the business functions and activities they relate to
• the identified essential characteristics of the records
• comparisons between original system functionality and target system functionality
• technical requirements of the original and target systems
• all system configurations, including metadata definitions and mappings
• all decisions, including decisions not to migrate certain metadata components of a record
• risk assessments
• any disposal and data clean-up performed prior to migration, including records of decisions and the processes undertaken
• management approval of migration strategies and plans
• any variations to strategies and plans
• technical reports on the migration itself, including date and time of the migration and all personnel involved
• details of all testing, pre and post migration, including comprehensive test reports which confirm the target system is functioning successfully and that all expected data and metadata has been migrated
• any necessary variation in records structure, design, metadata, format or content that will result or has resulted from the migration.

This documentation ensures the integrity of records, especially if needed for legal purposes.

See also AS 5393: 2025 Records and information management — Migration of authoritative data, information and records between systems

4. Retention and disposal of source records

Public offices must keep the source records after migration to ensure the migration was successful. Retaining the source records helps identify any issues that might arise after post-migration testing. The retention period should be calculated after successful post-migration testing and the validation of any issues.

Retention of source records

The retention period for source records should be based on a risk assessment.

Organisations should conduct this risk assessment in accordance with their internal risk management framework (as required by the Internal Audit and Risk Management Policy for the NSW Public Sector).

The Risk Management Toolkit developed by NSW Treasury provides advice on risk management processes and includes risk assessment templates.

This assessment considers:

• the business purpose and associated risk of the records
• potential consequences of lost or corrupted records
• the complexity of the migration and capacity of the target system
• the nature of metadata not being migrated.

For high-risk/ high-value records, it may be appropriate to retain source records longer. However the best way to guarantee that records are authentic, reliable, complete, and useable is to ensure that the migration planning, testing and documentation is robust.

The General Authority for source records that have been migrated (GA48) does not require source records to be destroyed after migration. Some public offices may choose to keep a reference copy for business needs or as part of the mitigation strategy in high-risk environments. However, if source records are retained, clear rules must be established to determine which version is the "official" record.

Retaining source records for long periods of time will not substitute for performing migration processes adequately.

Disposal of source records

The Standard on records management requires NSW public offices to document the disposal of records (requirement 3.7). NSW public offices must document the destruction of source records and metadata and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation might take the form of:

• metadata at the aggregate or system level which states that the previous version of this record group or system was destroyed on the specified date, by the identified authorised officer, in accordance with the conditions outlined in this authority
• metadata at a more granular level that documents the destruction of the previous version of specific files in accordance with necessary disposal requirements
• a final migration report that outlines the destruction of source records and metadata, providing enough detail to identify all records and metadata or groups of records and metadata that have been destroyed
• standard destruction approval forms used to authorise the destruction of records and metadata within the organisation.

NSW public offices must also document the transfer of records and metadata (e.g. to another organisation or, for State archives, to Museums of History NSW) and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation must identify:

• the records that have been transferred
• where the records were transferred to
• the date of the transfer.