First Boston Profits from IT Upgrade

Revamping their antiquated legacy database system yielded record trading profits
at CS First Boston, and cut costs by more than a million dollars per year.

In financial services, new technology is usually considered a cost center. The battle between IT professionals, who want to deliver leading edge systems and management, with its concern for the bottom line, is familiar to all. But what IT professionals know, but often can't demonstrate that technology redesign can have tremendous impact an the bottom fine - has been proven at CS First Boston.

CSFB reported a record $108 million profits in the first quarter this year, bolstered by a substantial increase in revenue from Fixed Income Trading operations. These results came shortly after the firm's Fixed Income Trading Technology group (FITT) completed a major overhaul of its trading database support systems, working with UNIF/X Inc., a New York City-based firm specializing in the design and automation of database management and support systems.

FITT serves all fixed income trading departments for CSFB's U.S. operations. The group is responsible for collecting and maintaining data from distributed trading operations, and ensuring its reliability.

When Steve Allen, a vice president of FITT and the senior person responsible for trading database support, came to work at CSFB in 1994, he encountered a number of significant technical challenges. These problems, like many industry-wide, lay with the firm's distributed database support systems.

When distributed client/server systems emerged as the foundation of next generation computing, CSFB downsized to Sun Sparcstations running Sun O/S. However, to leverage previous investments, CSFB, like many Wall Street firms, connected client/server systems to existing mainframes. Individual managers added hardware and software from different vendors as needed. Unfortunately, these pieces often didn't fit together as easily as the vendor's marketing materials promised. The complexity of financial databases also increased exponentially, as new financial products continuously developed.

"Because [CSFB] was relatively unstandardized, problems required much more time to fix," explains Sergey Fradkov, senior technical specialist at UNIF/X, who brought his experience working with distributed databases systems to the CSFB project, "And because things weren't configured properly in some cases, the system would crash more often."

Managing large database systems at the heart of the world's financial markets has never been a simple task. But maintaining and supporting very large relational databases in complex, heterogeneous environments now consumes a substantial part of every financial company's technology resources.

Surprisingly, trading database support groups are usually several times larger than a company's trading system development group. Before the revamp, CSFB had eight database administrators (DBAs), some of whom were expensive outside consultants. At an industry average of about $100 hour, Fradkov estimates that CSFB was paying about $1.5 million a year for support in FITT alone. Now the system is kept running with half the staff.

When looking at improving productivity, and cutting back on staffing, Allen found it important to have UNIF/X input. "I wanted [FITT] people to think about how to eliminate their jobs," says Allen, "which is not easy to get any employee to do, That's another reason why I relied heavily on Sergey."

PIECEMEAL CONFIGURATION

FITT's database support systems were plagued by a variety of problems, mostly relating to the fact that system components had been assembled piecemeal, over time. The nonstandard hardware and software configurations kept most maintenance and administration functions from being automated, and many processes, including data downloads and system backups, needed constant baby-sitting, unable to run successfully without manual intervention. Support costs increased steadily in tandem with the growing size and complexity of CSFB's databases - a tall order for any system support group.

System problems cut deeply into productivity. DBAs who took turns answering calls during the "nightcycle" were woken to troubleshoot so often they couldn't be productive during the day. And other DBAs spent more than two hours each morning investigating more problems from the previous night. Additional DBAs were brought in as consultants at a cost of almost a quarter of a million dollars a year each. Though the group's size continually increased, overwork remained a constant.

The company realized that these difficulties directly affected the updating and access of databases worth billions of dollars. As a first step to tackle the challenge, Allen and Fradkov analyzed existing hardware and software configurations, and developed a comprehensive plan to implement standards across all FITT systems. To save on maintenance and support costs, all database servers were set up to use standard directory structures. They also standardized disk and kernel configurations, as well as versions of database software.

The next step was to automate and increase the reliability of database maintenance applications, to minimize the problems occurring daily and during the nightly download of data from back office systems.
Fradkov and Allen developed a sophisticated system, controlled by an Autosys job scheduler, to completely automate database administration tasks. The new system reduced the number of problems, as well as costly nighttime support staff calls, by more than 85 percent.

Even after implementation of this, however, daytime DBAs still needed to find out if any issues from the night before remained unresolved. This involved a costly and time consuming manual double-check.

Allen worked with Fradkov to create a system to automatically diagnose and report those system difficulties. "We found it would be faster to develop software ourselves," says Fradkov. "We looked at third party software, like Patrol, and it required quite a bit of work to fit it into CSFB's environment."

Now, when DBAs begin work in the morning, they simply check their email for detailed messages about problems they need to troubleshoot. There is no need to launch a daily in-depth investigation to find the remaining issues.

To eliminate the need to baby-sit database backup processes, Fradkov recommended purchase of SQL Backtrack from Data Tools, which offers advanced data compression technology to reduce miltigigabyte databases to a size that is within a specified Unix file system's limits. "Data compression improves the stability of system backups across databases because it reduces the chance of running out of disk space while back-ups are running," Fradkov explains. "As a result, database maintenance applications also are less likely to fail."

While overhauling FITT's database support technology, Allen and Fradkov lived by this credo: utilities inherent in all systems should be generic and dynamic. A generic utility should perform many different functions. Dynamic utilities should be customizable with minimal effort.

They created a number of generic and dynamic utilities that can be used for multiple support and administration tasks. For example, a "permission manager" developed by the pair allows the business units to determine which users should have access to applications and data. "The best way for a DBA group not to be a bottleneck is to remove themselves where their involvement is not really appropriate," Allen says.

The impact of the technology changes made at FITT is ongoing. New systems have reduced the number of staff and consultant DBAs needed to support the Fixed Income Trading operation by more than 50 percent. By eliminating the most expensive consultants, and using less senior staff, FITT is saving over $1 million a year on headcount alone. Because systems are up and running regularly, DBA support turnaround time is lower. In effect, this "uptime" has boosted productivity - saving the company even more money. The remaining database support group can actually solve more problems for users, in less time.

Inside FITT, the work life of DBAs has dramatically improved and morale is higher. "No more sleepless nights for DBAS," notes Allen. "The redness is out of their eyes and the smiles are back on their faces." Record profits aren't hurting, either.

by Brad Mehl
Brad Mehl is a freelance writer in New York City.

   HOME   COMPANY INFO    SERVICES FIND US!

PARTNERS    |    UNIF/X IN THE PRESS    |    WEB/DB LINKS