According to Jeroen Kuijlen, CZ's I.T. Director, Dutch health insurance companies now face the same challenges as just about every other company in most other industries; listen to what your customers are demanding and then give it to them. Quickly!

“From now on, the choice of Health Insurance provider in The Netherlands will no longer be based solely on price. And the companies that eventually survive will be the ones that react quickest to customer demands...”

Already, CZ has seen an increase in requests for a faster and more comprehensive access to information on all aspects of health care. As a result, CZ has recently launched a new personal "My CZ" service for its customers, allowing them to go online and get instant access to key information, such as the status of claims payment. The system also lets them update basic personal information.

Via the public domain of CZ's website (www.cz.nl), non-customers and prospects can obtain online quotes for health care insurance, and optionally convert those quotes into real policies. It is also possible to check the lengths of waiting lists for specific treatment at hospitals in The Netherlands.

The majority of CZ's client data, which forms the basis of these new online services, is stored in a home-written application called M.A.R.S. This system was originally developed and maintained as a DATACOM database, but it has now been successfully converted to IBM's DB2 database system.

Jeroen Kuijlen explains the reasoning behind this move:
“The internet has now become a major sales channel. DB2 gives us far greater flexibility, because the information from the production M.A.R.S. DB2 database is now heavily used within our web applications. DB2's ability to interact with newer technologies, such as Java and Websphere helps us to develop services that keep us one step ahead of the competition...”

One example, using a Service Oriented Architecture with Webservices, allows health providers (e.g. hospitals, doctors, dentists, etc) to invoice CZ online for the services they provide to insured customers.

As well as the conversion of the actual client data from DATACOM to DB2, CZ also performed a migration of all its data to a new IBM Shark ESS 2105 Dasd Subsystem. Over 1Tb of data is now stored on the Sharks, 250Gb of which is occupied by the DB2/M.A.R.S. databases.

Another issue that had to be resolved before the new DB2/M.A.R.S. system went live was the method that would be employed to perform the daily backups of the databases. This project was headed by CZ Systems Programmer Peter Karel...

User Story

The Dutch health insurance industry will see some major changes in the coming years, and competition between the health insurance providers will be intensified.

Reducing costs and retaining the existing client base while at the same time attracting new customers, are just some of the challenges that lie ahead.

Health insurance providers will also need to have the appropriate systems in place to make it easy for new customers to switch between health insurers.

CZ is ready to face these new challenges. CZ is ready for the competition!

Its internal M.A.R.S. customer database, (based on IBM's DB2), is at the heart of CZ's future I.T. strategy. And as the importance of the M.A.R.S. system continues to grow, so too does the need to provide a reliable and secure backup & restore of the DB2 databases, without affecting the services now being offered.

This user story looks at how CZ has achieved that goal, utilising Innovation's FDR/ABR & FDRINSTANT.

DB2 Database Backup Issues

It was Peter's job to investigate the possible options for backing up the DB2 data.

“As a company, CZ was new to DB2, but we had numerous DB2 consultants on-site overseeing the development of the new databases and the eventual conversion of the data. Of course, those guys had their own ideas on how best to handle the backup and recovery of the databases, but I wanted to review all of the potential options.”

Peter was keen to take full advantage of the FlashCopy feature on the IBM Shark ESS 2105s. A similar Dasd Replication product had previously been employed by CZ with the daily backups of the old DATACOM M.A.R.S. system. After quiescing activity and creating duplicate offline point-in-time copies of the DATACOM disk volumes, those offline copies were then brought online and backed up by DFDSS, while access to the prime copies was handed back to the users again. However, as Peter explains, although this process helped to reduce the "backup window", there were several critical issues when it came to restoring data from the DFDSS backups...

“To get the duplicate disks online for DFDSS to back them up, we had to run a VTOC renaming utility. This then created discrepancies between the VTOC, VVDS and Catalog, which made restoring individual files (especially VSAM and SMS-managed) a real problem for us.”

If CZ was to employ a similar backup technique for the DB2 system (which makes heavy use of VSAM files), Peter knew he had to solve the problems on restores. The eventual solution came with Innovation's FDRINSTANT.

Utilising FDRINSTANT for DB2 Backup

CZ had already purchased FDR/ABR and FDRINSTANT, and Peter was well aware that the FDRINSTANT module could be used to backup offline duplicate volumes created by Data Replication products like FlashCopy, SnapShot, ShadowImage and TimeFinder, without the need for those duplicate volumes to be renamed and/or brought online before being backed up by FDR or ABR.

And FDRINSTANT had, in fact, already been implemented on CZ's non-DB2 data. But it wasn't until Peter was discussing the DB2 restore issues with an Innovation support technician that he realised it was also possible to utilise FDRINSTANT on the DB2 data as well. This facility was significantly enhanced with the DB2 LOG SUSPEND command (see below) at Version 7 (and Version 6 with PTFs).

Peter successfully tested the procedures and then put forward a proposal for the utilisation of FDRINSTANT in the production DB2 backup process. His next task was to convince the DB2 DBA consultants!

“At first, they weren't at all keen on the idea! To be fair, they were familiar with IBM's Image copy utility and they were reluctant to adopt a non-IBM tool to backup the DB2 databases. But once we showed them what FDRINSTANT was capable of doing, they couldn't wait to implement it!”

Interestingly, Peter says that some of those DB2 consultants have since moved on to new assignments at other companies where they are now actively promoting the use of FDRINSTANT with FDR/ABR for DB2 backup and recovery!

With the introduction into DB2 of the Log Suspend command, the benefits of FDRINSTANT with FDR/ABR can now also be employed in the backup and recovery of DB2 data.

After the Log Suspend has been issued to stop DB2 update activity, a combination of FDR/ABR and FDRINSTANT, together with hardware "replication techniques" (FlashCopy, TimeFinder, SnapShot, ShadowImage), can create a true backup of the DB2 databases in a matter of seconds. FDRINSTANT can then backup the offline DB2 databases to tape while full read-write access to the 'live' copy can be given back to the users (via the LOG RESUME command). If desired, the offline copies can be retained and used as input to an FDR or ABR 'instant' restore.

Now, with FDRINSTANT and Log Suspend, the DB2 backup window no longer runs into several hours. Instead, DB2 databases can now be fully secured with FDR or ABR without having to close DB2.

Restoring DB2 Tablespaces

As you may recall, aside from the benefits provided by FDRINSTANT on the DB2 backups, Peter's main goal was to improve and simplify restores. And he and his colleagues are very happy with the flexible restore features offered by FDR/ABR, especially when restoring individual DB2 Tablespaces (see side bar).

Johan van den Hout is one of the CZ Storage Administrators who is responsible for running "on request" DB2 restores:
“It's so simple. I just run a regular FDR/ABR restore of the DB2 tablespace(s). Because the FDRINSTANT-driven backups were taken directly from offline duplicate volumes, I don't have to re-label disks, or rename datasets. I just restore the data and let the DB2 guys issue the START DATABASE commands.”

Disaster Recovery

Johan is also one of the team responsible for CZ's Disaster Recovery implementation, which is tested regularly at an IBM recovery centre in the west of The Netherlands.

“It is a straight-forward and effective DR plan. We restore our
1-pack MVS system with Innovation's Stand Alone Restore (SAR). Then we restore all our Dasd volumes (including the DB2 disks) with FDR/ABR. Once everything is restored, we IPL our full system and away we go!”

Everyone Is Happy!

Peter Karel is happy. With the combination of FlashCopy and FDRINSTANT he can now ensure a regular daily backup of the DB2/M.A.R.S. system with the minimal disruption to services.

Johan van den Hout is happy. In the event of problems, he knows he can quickly and easily restore individual DB2 databases, or the entire DB2 system. And this can be done without requiring specialist DBA skills.

And Jeroen Kuijlen is definitely happy! He can clearly see the benefits of the technical advances made by Peter, Johan and the rest of his team.

“The DB2/M.A.R.S. system is at the very heart of the services we are now providing to our customers. It is essential that we fully secure the data on a daily basis.”

“It has to be backed up with minimum disruption to services. And in the event of local problems or a full disaster, we have to get the system running again as soon as possible.”

“I'm confident that we can do all those things with FDR/ABR and FDRINSTANT.”

Tablespace Recovery

Using a variation of the DB2 RECOVER command (RECOVER LOGONLY) allows applying of just the Log records to a Tablespace that has already been restored outside of DB2 control-i.e. with FDR/ABR.

1. First, issue the STOP DATABASE SPACENAM DB2 command to stop the Tablespace(s) to be restored.
2. Then, restore the Tablespace from the ABR backup system, by providing it with the MVS dataset name. ABR will automatically locate the most recent copy of this dataset, and then restore it. Note that if a Tablespace spans more than one DB2 linear dataset, all of the datasets involved must be restored to a consistent point-in-time.
3. Then, issue the DB2 START DATABASE command, with the ACCESS (UT) and RECOVER TABLESPACE (LOGONLY) operand.
4. Finally, issue the DB2 START DATABASE command with the ACCESS (RW) operand to restart the tablespace for full read-write access.

With the above method, recovering DB2 Tablespaces with FDR/ABR is significantly faster than Image copy. The restores are quick and easy and they can be done without specialist DB2/DBA skills.