Transaction Backup

From CometWiki

Contents 1 Transaction Backup/Checkpoint/Rollback 2 Backup 3 Checkpoint 4 Rollback 5 Added Benefits

Transaction Backup/Checkpoint/Rollback

Transaction Backup is both simple and complex.

Backup

The server will record every packet that comes to it that modifies data. There are a finite number of these packets that the file server receives:

• Create a File
• Create a Key?
• Erase a File
• Rename a File
• Delete a record
• Write a record. (includes all variants such as write, rewrite, update, insert, etc.)

These "transactions" will be written to a file. Each transaction will have a serial number which just increments each time a transaction is written. The transaction file will be "cycled" every so often. That is, a new transaction file will be created with its name containing the next transaction number. Cycling will be triggered by some time increment or some number of transactions. This will produce potentially many transaction files, but will allow them to be transmitted from server to server relatively easily.

• To be decided:
  • What is the file type?
    • Text file with variable length records
  • Where does it live?
  • What does it contain?
    1. Header Length - 4 bytes. Decimal number of bytes before the raw packed data. (includes these 4 bytes)
    2. Transaction number - up to 8 bytes (Decimal)
    3. Time Stamp - Local date and time of the transaction. YYYYMMDDhhmmss
    4. Partition Number - up to 4 bytes (Decimal)
    5. Operation - up to 2 bytes (Hex)
    6. Node Name -- up to 16 bytes (Quoted)
    7. Program Name - up to 8 bytes (Quoted)
    8. File Path - up to 260 bytes (If Keyed File, Points to the data file) (Quoted)
    9. Unpacked Data Length - 4 bytes (Decimal)
    10. Packed Data Length - 4 bytes (Decimal)
    11. Packed Data -- Depends on operation -- up to 8194 bytes -- could contain the Key
• All Header fields are separated with commas.
• Transaction records end with CR/LF.
• What about encryption? Is this needed? If so, at what point in the process?
• What about a time stamp? Is this needed?

Checkpoint

Periodically, the server will perform a "checkpoint". That is, it will make a copy of all data which is valid up to a certain point in time to a backup folder.

This operation could take some time and provision must be made for users to keep working during this process.

The following procedure allows this while providing a "snapshot" of the files as of one instance in time.

There are at least 2 ways to accomplish this

1. Always make a full copy of all of the data. This snapshot will be at the end of the process, not at the beginning. How important is this?
   1. Remember the next transaction number (lets call it 1111)
   2. Start copying files from the main directories to a backup folder.
   3. When copying is finished, remember the last transaction number. (lets call it 2222)
   4. "Play" the transactions to the files in the backup folder from the first one to the last one (1111 to 2222)
   5. Compress all of the backup folder and name the compressed file using the last transaction number it contains (in our example, 2222).
2. Use the Transaction Logs to make the snapshot. This method requires another copy of all of the files to already be established in the backup folder. This method will perform better and guarantees that the checkpoint will be up to date at the instant the checkpoint is initiated.
   1. Remember the last transaction number (lets call it 1111).
   2. Look up the transaction number for the previous checkpoint (Lets call it 0000).
   3. Start a new transaction file (1112).
   4. Remember this transaction number for the next checkpoint operation (1112).
   5. Playback all transactions from 0000 to 1111 to the backup folder.
      • If the destination file does not exist for the transaction, copy the file from the active folder. There may be some danger in this.
   6. Compress all of the backup folder and name the compressed file using the last transaction number it contains (in our example, 1111).

Compressed Checkpoint files could be retained, in an archive.

At the moment, I am favoring method number 2. --Jim 00:20, 15 September 2009 (UTC)

Rollback

At any time, the user can "rollback" the whole system by restoring a checkpoint and playing transactions up to a desired point in time. This could be on a completely different system than the main server. This would facilitate migration to a new server, making a test system, or whatever.

Added Benefits

There are several added benefits from this feature

• The Disaster Recovery Service could work with transaction files, potentially speeding up the whole process. Transaction files could be transmitted many times a day making the whole service more valuable.
• Developers could analyze transaction files to determine just how some record was modified etc.