Agile Adoption by the Financial Services Industry

Agile Adoption by the Financial Services Industry

Seeing a Greater Return on IT Investment

Today’s financial institutions are facing huge changes in technology platforms, payments processing systems, financial systems, asset and risk management systems, while attempting to deliver services in the way customers prefer. From m-payments and the ability to view and trade stock options via mobile phones, to e-payment and trends towards an increase in digital and online banking, to the need to rapidly process and keep track of accounts, balances, interest rates and identify financial trends, while reducing financial risk, the platforms and business applications banks and other financial firms use have evolved enormously in recent years, and are continuing to do so at a rapid rate.

Firms in the financial sector also face other challenges, including the need to comply with regulations such as Sarbanes Oxley, SEC, FDIC and the Federal Reserve, while if international, they may also require compliance with Basel II and SEPA directives, among others.

Combine this with the difficult market conditions in the past few years, and the need to sustain revenue growth and retain market share in an increasingly competitive landscape, and IT departments within financial firms are feeling increasing pressure to improve efficiency and speed, while at the same time maintaining controls on cost and capital outlays.

Agile development methods meet many of these requirements. The trend to Agile adoption by other business sectors has been driven by the need to deliver high value, create software that actually meets end user goals, and to reduce risk when business applications are developed.  Yet when financial institutions think “agile”, traditionally they have reacted with “hesitate.”  This has occurred for several reasons, not the least of which is the desire for absolute predictability of costs in a sector that spends large amounts on its IT.

Challenges Faced by Financial Institutions: Big Spending on New Services and to Maintain Aging Systems

The financial services industry is a large consumer of IT services, with its IT spending in North America expected to reach $71 billion, and to continue to grow at a four year compound annual growth rate (CAGR) of 4.1 percent¹.  Global spending will also continue to be significant, with global spending by financial services on IT predicted to reach $393 billion by 2013, with Europe and North America spending the lion’s share of the total (69.1 percent).²

This is a lot of money going towards IT, but why? One reason is the rising demand for convenience banking, with online and mobile banking two of the fastest areas of growth.  But the large investments in IT are also due to a less forward-looking factor that heavily impacts the banking industry. Many financial institutions and banks today have a large and aging legacy IT infrastructure. They originally heavily invested in during the late 90’s, and now these networks require large amounts of patching, maintenance, upgrades and continuous attempts to integrate it with new technologies.

In addition, larger banks are tending to buy out smaller independents, and are inheriting older systems that they must either choose to work with, or completely revise.  This choice can be complicated if a core business platform is surrounded by layers of maintenance code.

Barriers to Adopting Agile within a Financial Institution

Because of the large amount of regulation within the industry, banks have often continued with waterfall methodologies, or “the tried and true,” due to its perception of having more predictable, defined outcomes. In an industry driven by cost analysis, the ability to define each requirement and its cost before starting development has made it attractive for years to this sector.

The challenges of Agile adoption by financial institutions is often greatest within the largest firms, which must coordinate development between segregated teams that work individually on one component of the software, such as design, analysis, development, or testing.  These teams may be distributed across different time zones or even continents.

Another reason for the hesitation to adopt Agile is the fact that in financial services applications, even the smallest error can cause the loss of thousands or millions, especially when account sweeps and trades must be timed with extreme accuracy. The technology cannot fail, or the risk exposure is unacceptable. The requirements for due diligence have caused many financial services firms to stay with waterfall development, where the documentation is extremely extensive, for compliance with auditory and other regulations.

Because of this regulation, the financial services industry has traditionally utilized very formal enterprise change control procedures, and created extensive documentation for auditing purposes. In an industry where “risk management” is the order of the day, any development methodology that has a perceived lack of outcome predictability is not quickly embraced.

Another challenge is the nature of financial software systems, which may interface with multiple peripheral systems and interfaces, requiring careful definition of these interfaces ahead of time. These complex systems can be difficult to develop using a pure Agile methodology, and some, like embedded software, may be more suited to waterfall or a hybrid development process.

Yet in spite of these challenges, a growing number of financial services firms are choosing to transition to Agile. This is due largely to the problems seen – and the costs associated – with staying with waterfall methodologies.

Problems Financial Firms Encounter with Waterfall Development

 

Traditional waterfall development practices have caused difficulties for numerous financial firms. One that has experienced issues with it is one of the leaders in the financial industries, whose Capital Finance group is currently exploring making the transition from waterfall to Agile and other methodologies. They are looking for a lighter, more responsive way to develop the numerous smaller projects in-house which the time and resource-intensive practices of waterfall are not really suited for. Currently, cPrime is working with them on training in Scrum and Kanban methodologies, providing coaching and mentoring for some initial pilot projects.

The issues faced by Farm Credit Services of America when they used waterfall is another example of the problems encountered by financial services firms that stay with waterfall. Farm Credit Services is a cooperative association that grants loans to Midwest farmers and ranchers, to the tune of over $11 billion of operating capital and real estate financing.  For years, they followed a traditional waterfall approach, and saw customer satisfaction fall as a result. They would spend a lengthy time gathering requirements, carefully defining each, before even beginning development, which frequently lasted for years.

In the meantime, technologies, business practices, and even the original business partners who requested the application would change. And many times, the customer wasn’t happy with the end result, since by the time development was completed, it needed significant changes to keep up with the changes in service delivery and technologies.

This caused Farm Credit Services to decide to change over to Agile development. During the transition, they formed an Agile Champions Team (ACT), and their six development teams now use Scrum. Since the change, they have seen a significant reduction in software defects, and more rapid development times. Most importantly, they have seen a rise in customer satisfaction and development team morale³.

This quick case study highlights some of the problems seen with waterfall. Financial institutions that choose to use it to develop strategic projects often see these projects come in late, and over budget.  In fact, delays in the anticipated launch dates for financial services projects have become notorious – and even expected – due to the lengthy process that waterfall requires.  This causes an industry that concerns itself with costs and the bottom line watch project costs rise, as teams struggle to rework applications that must attempt to keep up with changes in technologies and platforms – even though the requirements and specifications, which took months to obtain,  were defined two or three years previously.

Customer satisfaction with the final applications may be lower with waterfall, as noted before. Even if the application fulfills all the features requested, the customer must be an extremely patient one to deal with waiting two or three years to see a finished product.

Development team morale may be reduced, since developers must rush to test and re-test at the end of a project, realizing that they are working on an application that is already outdated before it even goes to market. And time and resources are often wasted, since some of the original features specified and built into the software are never used, once the application becomes operational and actual use shows that certain features aren’t needed or wanted.

Because of these waterfall drawbacks, financial services firms are turning – albeit slowly-towards investigating Agile development methodologies. They want to reduce time-to-market, and see the improved ability to incorporate customer feedback that Agile provides.

For instance, European-based BNP Paribas operates in over 85 countries, and in 2006 reported a net income of 7.1 billion Euro. In order to improve development team quality and reduce development time for projects, an initial team of 60 developers in the risk IT department adopted Agile for new projects, beginning in 2004. The teams consisted of six developers each, who were trained and mentored in Agile, and they provide working software features during iterations of two weeks’ duration, to incorporate stakeholder feedback.  They also use test-driven development, and the project velocities have improved greatly⁴.

Capital One, a Fortune 500 firm, offers financial products that include credit cards, savings accounts, and loans for consumer and business purposes. They first adopted Agile in 2004, and after moving to Scrum, by the end of 2006 saw an average of 70% reduction in time-to-market, causing them to implement Agile across their enterprise in 2007. By doing so, they also so improved collaboration and teamwork in the development/Scrum teams. Agile worked so well, that they then began developing internally Scrum coaches who could oversee Agile development within the firm⁵.

cPrime has assisted numerous clients in the financial industry make the transition to Agile. (See the interview with cPrime Agile Coach, Jeff Howey, below)

These firms are leaders in the financial industry, and they have seen the advantages of Agile for their organization in terms of improved ability to incorporate feedback, flexibility, improved velocity and high quality in the code produced.

The Key: Transition at a Workable Pace

When a financial service decides to go Agile, it makes sense for the transition to be implemented in stages. At cPrime, we recommend this approach, making small changes at first, that will realize the most value. This is why the planning stage is so important. Careful planning can make the difference between Agile adoption that lets the firm enjoy all of the benefits, or one that falters.

At cPrime, we meet with key decision-makers and managers during an Agile assessment, to ensure consistency and that all are in agreement before going forward with the transition plan. Prior to the transition, the tools to be used are selected, and/or developed and adapted for in-house use.  This is an important part of the transition, since the documentation the tools provide can be critical for a firm that must comply with regulatory and auditory requirements. These can be combined with templates to help automate many of the processes, and provide a documentation trail for later use.

Once the tools are selected, the developers undergo careful training in Agile and Scrum practices, with on-site coaching and mentoring by cPrime staff who have extensive industry experience in Agile implementation within Fortune 500 firms. This training, coaching and mentoring, with actual hands-on practice in using Agile for “real life” development scenarios is critical to a successful Agile transition. Our firm strongly believes that it’s important that all team members understand, and are shown how to use Agile in a practical way, instead of “guessing” how it’s done.

First, the Scrum teams are trained in their roles. This includes Scrum Masters, who help facilitate development, Product Owners who act as intermediaries between stakeholders and development teams and update product backlogs containing the list of the features desired in the finished product) and the development teams. Team leaders learn to enter features into the Sprint backlogs during the transition time. Then, they begin to take on projects: first, small pilot projects are often completed; then initial success is built upon, with larger ones developed over time. Key elements that are different from waterfall include the extensive collaboration, since teams are often multi-disciplinary, and the increased communication, with daily Scrum meetings to asses where development currently is, where it’s going, and identifying any issues or roadblocks that come up.

The Sprints, or time periods for developing specific features in a product backlog, are often shorter than iterations in waterfall, with a Sprint lasting only two to four weeks. The team leaders are taught to estimate the time required to complete features, and to allocate resources during a Sprint, with mentoring and modelling by cPrime staff for the transition. These estimates become more accurate with each Sprint completed. Burndown charts are created that show the backlog items completed to date, and the team velocities are graphically shown to provide visuals of the progress.

Sometimes a Hybrid Approach Makes Sense

For some financial institutions, a hybrid approach may work best, especially at first. cPrime has worked with numerous clients in the financial services sector. The best results occur when management supports the transition, and both the stakeholders and the teams doing development begin to see the benefits quickly, through a carefully planned and executed transition process. Defining whether a hybrid approach will be used – and when – is also done, by mutual agreement.

See cPrime’s Hybrid Agile Model Workshop

By using these methods, the transition to Agile for a financial services firm can be an excellent experience, and the company can begin to enjoy the advantages of Agile: improved velocity, improved ability to incorporate customer feedback; improved collaboration, teamwork and morale internally, and greater customer satisfaction.  The companies using Agile are better able to stay competitive, and use the evolving technologies within the financial services sector. And in the end, they are able to keep pace with marketplace realities, and provide the services that customers require, on the platforms they prefer, with a shorter time-to-market and a greater return on their IT investment.

 

Interview with Jeff Howey, cPrime’s Agile Coach, around the barrier to adopting Agile as a financial industry.

1. What are some of the barriers to adopting that you have seen, from financial services firms? How have you addressed these hesitations?

Intentionally detailed and methodical approaches to managing requirements (often driven by internal Audit or SOX rules) tend to attract very detail-oriented and methodical people into project management, leadership and customer-facing positions.  Agile requires the ability to work with “good enough” requirements and agreement.  Forcing a process to approve all requirements and manage every step of the process (often at a micro level) is anathema to Agile.  To varying degrees of success, this depends on the ability of the teams to prove the value of Agile delivery and maintain quality in their releases while also satisfying the requirements of the process.  I have seen many Project Managers or leaders of organizations who would be called “micromanagers” in some circumstances become major proponents of Agile when the teams did not ignore the process (or complain excessively about it) and deliver a few releases of high-quality software in frequent release patterns.

Many financial services systems are large and complex with many integration points.  And they deal with money.  This requires extensive testing.   In most of my experiences, the teams would perform Stabilization Sprints to do extensive testing and validation in advance of a release.  This would be particularly true where multiple teams were working on different components of a system that would require integration and stabilization.

2. What are some specific problems with waterfall that cause financial services companies to finally change over from waterfall to Agile?

Customers who used to think they had no choice but to wait months or years for updates to their systems have gotten savvy.  And so has the competition.  In addition to that, many organizations themselves have seen that big projects with a long shelf-life tie up a large amount of investment.  While the pace required to keep up with customer demands and competitors’ innovations has increased, these organizations understand that focusing that investment on short time intervals and having the ability to pivot quickly and gracefully is supported with Agile and forces the organization to choose the most important projects (and features of each project).

3. What helps the transition most? What makes it hardest?

Helping a transition MOST is to have understanding and of the acceptance of the trade-offs of Agile by all levels of an organization.  For example, starting early and working iteratively to deliver requirements and demonstrate them to the Product Owner is intended to make it possible for the team to work collaboratively and quickly toward releasing a valuable product.  It may, at times, unintentionally put the team into a position that future releases require rework.  For instance, a first release of a system may satisfy most needs of most customers and be releasable in a short amount of time.  The goal of the team is to satisfy the remaining needs of remaining customers in the future (as trying to accomplish all of that work would greatly extend the time to deliver).  Most of the time, the requirements  for future releases would be known and taken into account through the architecture and approach of the Release 1 deliverable.  At times, the team may choose to do “just enough” to get the first release out the door and intentionally plan to do the additional work in the future.  That isn’t a huge issue, though it does create work.  The hardest point is that sometimes, it is the smaller details that are unintentionally overlooked or made apparent only through very close inspection and understanding of complex rules or integration points that cause the most pain.  When this is understood at all levels, the organization can adapt gracefully.  When this is not understood, it causes pain and finger-pointing.

There are many obstacles to a smooth transition.  In many cases, it is the resource-allocation model that does not allow a Scrum team to stick together and be dedicated to 1 project together for long periods of time.  That, as an organizational policy, is the hardest thing to overcome – and one of the things that makes a transition move smoothly.  Even harder to overcome is a transition to using Agile processes to deliver software when the rest of the organization (particularly those who are Customer-facing) do not understand or buy-in to the process.  Or, if those organizations have pushed the delivery organizations to use Agile with unreasonable expectations (e.g. Agile does NOT mean “do more with less, faster” – it is about intelligently making frequent tradeoffs to get the most business value sooner.

4.How long does an average transition to Agile take?

This is a tough question… I don’t know that any transition is “average”… but I’ve seen transitions take anywhere from a few months to “never”…

I worked on a pilot project with a Property & Casualty Insurer that went well.  Within a year, a handful of other projects were following Scrum and it was an option within the PMO approaches to choose Scrum for some projects based on select criteria.

I’ve seen attempts by several organizations, including one large banking institution, to move toward Agile as their preferred method of delivery as many as 5 years ago who are still struggling to manage the transition and implement Scrum successfully.

I’m aware of one client who adopted some basic Scrum practices to give more structure to their process.  For years, they were driven by “fire drills” or demands their largest customers in a very reactive environment where customers were driving the dates.  The team was a shared resource from the point that they developed for new requirements but also had to do all of the ongoing support.  The technology team was unable to get the customer-facing organizations on board (as it would require prioritization, scheduling and, more importantly, saying “no” to low-priority customer requests) to manage the shared Backlog with a small team of developers.  This team abandoned Agile and went back to fire-drill mode, overtime and chaotic prioritization as they were unable to stick to Sprint or Release schedules in that environment.

But, mostly, I’ve seen that teams who work together for long periods of time and can focus on 1 project at a time develop mastery of Scrum or Kanban within 4-6 months and are able to really focus on continuously improving their ability to work together and deliver predictably and reliably.  In many organizations, this sets the stage for Agile adoption as one of the approaches for projects.  Where the organization is able to implement structural and procedural processes in support of Agile (e.g. resource allocation models, portfolio management, backlog prioritization, etc.), a widespread adoption of Agile and true Transformation can begin.  In many cases, it is somewhere between the first and second year of multiple projects running in an Agile process that this occurs.  The size of the organization is certainly a part of the issue, but the biggest driver is culture at the team or department level.

5. Do you use a hybrid approach? What kind? Which departments tend to go Agile, which ones tend to stay waterfall?

In large organizations with intentionally heavy project management methodologies or for complex and large, highly-integrated systems, hybrid is the usual choice.  For some teams, Agile is not ideal due to their resource management approach, so a plan-driven approach works best to plan the work.  For some systems, iterative requirements and development are not ideal or do not have support of the business organization and must be written far in advance for approval.

In terms of trends toward Agile, any organization with web or mobile applications tends to move quickly toward Agile, particularly Scrum.  Business Intelligence, Production Support and digital marketing organizations tend to move quickly toward Kanban.  Infrastructure, Data Warehousing and some of the back-office systems tend to continue development in a waterfall approach.

6. What key points should be addressed in Agile adoption, in your opinion?

Culture of the immediate team and organization, not size of the parent company, is the key determining factor in whether some products can be delivered using Agile.

It’s OK for some projects to use Agile while others do not…

Well-documented and well-managed projects (conforming to PMO and other rules, including SOX regulations) can follow Agile… it’s all about how we collaborate with the customer… and less about the speed with which we deliver.

Sources

1. 1. IDC – Worldwide Vertical Markets IT Spending 2010-2015 Forecast: 1Q11 (March 2012)

 

1. 1. Jegher, Jacob. “IT Spending in Financial Services: A Global Perspective,” Celent Research report, available at http://www.celent.com/node/26799

 

1. 1. Wiss, Urs.  “Agile Software Development in the Finance Industry: How popular are agile methods in financial services firms.” Bachelor’s Thesis, 2/17/2008

 

1. 1. Saran, Cliff. “European banking giant adopts agile development methodology,” ComputerWeekly.com, 11/26/2004. http://www.computerweekly.com/feature/European-banking-giant-adopts-agile-development-methodology

 

1. 1. Silva, Kara and Doss, Chris. “The Growth of an Agile Coach Community at a Fortune 200 Company.” Capital One Financial Corporation, IEEE publication ©2007.