Leaders weigh in on Titus’s new DevOps culture that embraces quality and security.
Quality and security are not just buzzwords when it comes to Titus’s software development priorities; they are the driving force for how the company develops its data classification software and brings it to market.
Anna Lynch, VP, Engineering, and Connie McFarland, Senior Enterprise Software Architect, together with engineering managers have led the charge to create an inclusive DevOps culture.
The company’s 55 developers and software architects in Ottawa and Toronto now embrace a common set of values that include regular architecture and quality reviews at each phase of the development and testing.
This initiative is backed by senior executives who also have rolled out security and quality training across the community.
Thanks to this cultural shift, Titus has continued to enhance the quality of its software product launches, including the company’s recent Software as a service (SaaS) cloud-based phased rollout.
“At the end of the day it’s a customer win because they are getting higher quality products and we’re able to better support them in their requirements,” says Kosta Hatzis, vice president, Product Success, at Titus.
With the SaaS model and Titus’s DevOps capabilities, Hatzis’s team can bring Development in faster than ever before to resolve priority issues.
“Over the last three months we have seen a 29% decrease in time to resolution of cases,” says Kosta. “Customers are seeing greater success with our products and because of that, we don’t see as many entry-level issues surfacing.”
Below, Anna and Connie share what makes their approach unique and highlight best practices for creating a quality-driven software development organization that will lead to more productive teams and ultimately, better products.
Q. How are quality and security integral to Titus’s software development process?
Anna: Because Titus is part of a security product, we must make sure that when we deliver software, we’ve built it with security in mind. It’s everything from how we code to how we package it up, to how we test it.
If I look at just quality alone, the worst thing that could happen is we deliver a product and customers find bugs.
The more we find bugs during the development process, the better it is because it means the less our customers will find and more important, experience a smoother deployment and seamless product use. So, we do everything we can within the product to make sure that we adhere to quality.
At the beginning of a product cycle we create quality plans and set targets for everything from the number of bugs to the performance numbers to the features themselves and what we did to test them.
Every two weeks the team reviews what they’ve accomplished, including reviewing the quality meter. If they hit a threshold of a large number of bugs, we may not proceed until we fix the problem.
The more you go into the development cycle, the more bugs you add. If you don’t fix them at the time when you’re building it, it only gets worse.
And they’re harder to fix at the end of the cycle than at the beginning.
What we’re really getting at is ‘How are you managing your defects?’ During the two-week period we actually implement a bug day. It’s all about building up the test environment that’s needed to make sure that that code is strong.
At the end of the release, teams are required to fill out a quality certification where they compare their quality targets to what they’ve actually achieved. If the bug counts or performance are not where we like it, we can reject the release.
We must make sure that when we deliver software, we’ve built it with security in mind. It’s everything from how we code to how we package it up, to how we test it.– Anna Lynch, VP, Engineering, Titus
Connie: Before we even start coding, we institute Architecture Reviews, to make sure that the architecture holds together.
We have an Architecture Council to ensure that it’s not one person deciding on a piece of code, but it’s a team of architects saying, “This makes sense”, or “This doesn’t make sense”, only after this process does the development team start coding.
Q. What has been the impact of that approach? How has it resulted in a better software product for customers?
Anna: Bringing this council together to review all the options has really made a difference because we now make the right business decision, not a personal decision based on a developer’s preference for a tool or feature.
Q. Has getting buy-in on the architecture up front made a difference with a product launch?
Anna: Our SaaS product – which embraces a subscription model to deliver our data services in the cloud – benefited from the Architecture Council.
This was by far the largest Titus product that we ever put together because we had both the classification and the identification piece and the total infrastructure around it. It required strong coordination – we had to pick the right tools because one product had to speak to the other product.
We are taking a phased approach for the first year, but this is the model for how we will deliver our software going forward.
Q. Can you be too close as a software engineer to validate your quality checks? How do you engage with customers to ensure the quality of your software?
Connie: We are all about validation – we have our product success team that discusses our product roadmap with customers, and we’ll get their feedback.
We also have defects that have been reported by customers. Those are probably the higher priority defects to fix. We’ve worked diligently to get our customer-identified defects really low. We have targets to make sure that we fix a certain number of customer defects for every release.
We also are very rigorous about testing – we do both white box and black box testing.
Developers do the white box testing. Every time we put new code in system, every single addition to the system has to be code reviewed. Once it goes into the system, the system lets you know if you have introduced code that is a bug, because these tools can tell you this condition. You can put gates in place that say if it drops below a certain level, you can’t put that code in.
We are all about validation – we have our product success team that discusses our product roadmap with customers, and we’ll get their feedback.– Connie McFarland, Senior Enterprise Software Architect, Titus
Q. Is Titus ahead of the industry in its software development quality efforts? What’s been the impact?
Anna: We are an early adopter of what’s called the DevOps culture. For every step of the process, we have automated tools to help us understand the quality of the product.
Kosta Hatzis, who leads product success at Titus, calls our DevOps process a “great win” for our customers. He reports that call resolution times have improved by nearly one-third because of DevOps capabilities.
Shifting to a SaaS model has changed how Titus engages with customers in that “we take a swarming approach to priority issues, with software developers being brought in closer and faster,” Kosta says.
Traditional tiered support models no longer apply: “The whole SaaS environment brings everyone to the forefront to engage with our customers and that’s had a very positive impact for the whole organization,” he says.
Connie: Also, because of this process, we received our first pen test report where we had no critical defects. It was a third-party company that did the analysis.
Q. What are the key security tools that you use when developing new software?
Anna: We have tools that perform static scans to tell us if there’s any vulnerabilities. This is run against the code itself and we can stop a build if the scan is not clean. After we perform static scan, unit test and product build, then we run dynamic scan of any web pages being developed.
Finally, we do penetration testing, using white hat hackers. Titus hires a third-party company that comes in and tries to hack our code.
The static and dynamic scans run as the build happens. The penetration testing once during the development cycle and before release.
Q. How important is training?
Anna: We invested in online security training for our entire team. We built modules based on their function (software development, QA, architecture, etc.) and gave them badges when they finished the whole set, which they proudly displayed.
Q. Where do developers go wrong when it comes quality coding?
Connie: Inhibitors to good code for developers are usually two things: ego and optimism. Ego makes you think that you do not write bad code. And even the best coder writes bad code sometimes. And, then optimism, because you think, ‘Oh I thought of everything.’ Nobody can think of everything.
That’s why our Architecture Council is so effective. When code is a community affair, it creates accountability. When someone’s looking at your code, you can’t take those shortcuts. It helps you become a better programmer in general.
Anna: The other inhibitor is that companies won’t invest. Being willing to invest is important because then you have tools in place as well as transparency built in. That’s why it’s so important to have buy-in from senior management.
You must have support from the top. If senior management didn’t support it and make sure that people across the board followed, it would have been much harder to sell to the teams.
Q. Any final thoughts?
Anna: We take quality and security very seriously. Because if you don’t, your customer will. It will provide your customer with the confidence they need to ensure our products are running securely within their environments.
The best customer feedback is not glowing reviews, but that they have implemented our software in their environment and they’re using it.
Connie: I agree. To use an analogy of a well-made garment: You know that if you turn it inside out and the seams are also nice, that’s the security standard we have to meet.
Similarly, if you buy a cheap garment and pull on a thread, the thing falls apart. We need those seams to hold together under stress. Developers need the tools and support of their organization. Companies need to value the inside of that garment and how well it’s been constructed.
If I were making a purchase of Titus for another organization, I would want to be assured that the software was built for quality.
How are we handling their content, which is really sensitive?
Are we worrying about all the right things?
If I knew the numerous people involved, and the detailed quality process they took to produce the software, I would feel much better about making the recommendation.
Source:https://titus.com/blog/data-security/tituss-inclusive-software-development-process
