Early Adoption of SQL Server 2022! Outcomes and Feedback of the SBI Liquidity Market’s Next-Generation Trading System

Database renewal deliberated as part of the launch of a next-generation trading system project

SBI Liquidity Market provides platforms for exchange trading for SBI Group businesses such as SBI SECURITIES, SBI FXTRADE, and SBI Sumishin Net Bank. In FY 2021, the company’s foreign exchange transaction was 10.2 trillion dollars, the number of FX (Foreign Exchange Margin Trading) accounts were at 1.72 million, the most in the industry, and FX deposits were at 319.8 billion yen, which was also No. 1 in the industry. As a business on a major scale, the company also develops new exchange product services, and actively utilizes new technologies such as blockchain, AI, and big data analysis. As the company strived to expand their FX business in various categories, including promotion of overseas businesses, their aging FX system, which was the infrastructure of their business, became an issue. It had become difficult to swiftly respond to business requirements due to their dated system, which was constructed at the time of the company’s establishment and tweaked repeatedly over more than a decade of use. Yuta Yoshikawa, System Development General Manager at SBI Liquidity Market, provides a backdrop of this project, explaining that in addition to solutions for issues, there was a need to introduce a next-generation system that also boosted performance.

“Although we had been conducting peripheral improvements to the system, we had not been able to tackle the infrastructure itself for a very long time. These continuous sporadic patch-ups resulted in the system not being able to respond to recent business demands in a timely manner. As the scale our business grew, ensuring the wanted business performance became impossible. To resolve these issues, we launched a project in October 2021 to introduce Next-generation Trading System” (Mr. Yoshikawa).

In this way, the SBI Liquidity Market’s Next-generation Trading System project started up, with Mr. Yoshikawa, who took the position of project manager, at the helm. Four goals for this project were established: “Pursuit of low latency and high availability”, “Swift business deployment”, “Optimization of total costs”, and “an Architecture that looks toward the next decade”. “The goal was to establish a fast, safe, easy-to-handle, cost-effective architecture that can be used for a decade or more without any issues,” Mr. Yoshikawa explains.

As deliberations for the database structure went on, Mr. Yoshikawa says that he received a proposal from Microsoft. With this, he says he started to consider adopting SQL Server 2022, the latest RDBMS (Relational Database Management System).

“We had been using SQL Server in our conventional system. We were thinking about adopting the previous version, which had an impressive track record of use. However, Microsoft, which had been regularly exchanging information with us since that time, proposed that we introduce SQL Server 2022, which was the latest version. As the version had not been released at that time, we joined the EAP to make further deliberations, and decided that the system could achieve our four goals, which led to the system’s adoption” (Mr. Yoshikawa).

Sungho Hong, of the SBI Liquidity Market System Development Department, was the leader of the database build team and responsible for deliberations of the early introduction of SQL Server 2022. He explains the history of this early introduction and the deciding factors of SQL Server 2022.

“We had been using SQL Server since our company’s establishment. The development department, and the management and business departments as well, considered the system as a reliable one. Our participation in EAP saw results that exceeded expectations for our required performance and functions, which became the deciding factor for adopting the system” (Mr. Hong).

Application of advanced functions such as Always On availability groups and intelligent query processing, along with a system composition with DR site construction in mind, using Azure SQL Managed Instance

SBI Liquidity Market felt confident about the EAP test results and decided to adopt SQL Server 2022, and will continue to construct its Next-generation Trading System as an early-adoption company with the support of Microsoft. SQL Server 2022 has enhanced features to connect with clouds (Microsoft Azure, hereafter Azure). For this project as well, he says that they strived to attain both availability and performance by combining the conventional on-premises environments with cloud technology. According to Mr. Yoshikawa, as a challenge in their development of Next-generation Trading System, they constructed a mechanism to run applications in a secure environment by using the AKS hybrid deployment options that can execute Azure Kubernetes Service (AKS) on premises.

“AKS, which is the Kubernetes platform offered by Microsoft, is commonly used as a managed service on public clouds (Azure) in most cases. However, as our project is a system to be provided to financial companies, it has been constructed to be installed within an on-premises environment that is managed by us. In detail, we use AKS hybrid deployment options to create a virtual server on a Windows Server, on which an AKS cluster is created to run the application on a container basis. The database of the system also adopts a framework that executes SQL Server 2022 on a Windows Server to construct a mechanism that operates as an FX platform. However, AKS is fundamentally a service that is assumed to be operated on Azure, so it has diverse collaborative functions with Azure. For this reason, we applied services such as Azure Active Directory (Azure AD) and Azure Container Registry to our project as well, so in this sense, we are aiming for a hybrid framework that is both on premises and on clouds” (Mr. Yoshikawa).

Mr. Hong, who is involved with the development, operation, and maintenance of the existing system, mentions adopting the Always On availability group as a device to build the system.

“The database for our Next-generation Trading System consists of three physical servers in on-premises environment to create an Always On availability group. For the future, we are also thinking about building DR sites on Azure side using the link functions of Azure SQL Managed Instance” (Mr. Hong).

Mr. Yoshikawa explains, “By adopting Always On availability groups, we have achieved higher availability than our conventional system, which was built with failover clusters to ensure availability.” He stresses that, “We were able to greatly improve availability by significantly shortening switch-over times to standby systems when the operating system ran into trouble, or as we call it, at failover.” For operational load reduction and performance enhancement, he also says that the IQP (Intelligent Query Processing) function, enhanced in SQL Server 2022, also had a significant effect.

“IQP functions, which create feedback for what had been executed in SQL Server to redo a new plan, were very effective when creating execution plans. Our trading system has over 10,000 transactions every second. Some customers may trade a lot while others may not trade as much, and some trade only in dollars and yen with no other currencies. In this way, we had an uneven data concentration to work with. Due to this, there was a need to create execution plans after checking for optimization at each execution, or else there may be a major decrease in performance according to the timing. Our response to this issue in the previous system consisted of various efforts made in operation, but for this time’s project, we have achieved a significant reduction of operational loads by utilizing IQP functions” (Mr. Yoshikawa).

Ensuring business continuity and reinforcing security by actively utilizing collaborative functions with Azure

Furthermore, by adopting the latest SQL Server 2022 for this project, they were able to reinforce connections with public clouds (Azure) as well. As mentioned earlier, SBI Liquidity Market says that have DR site construction that utilizes the link functions of Azure SQL Managed Instance to preserve data on Azure in mind. With this, they expect to ensure business continuity in the event of issues occurring at their on-premises environment. Mr. Yoshikawa says that applications that run on the FX platform constructed on-premises using AKS hybrid deployment options and SQL Server 2022 can also be executed easily on Azure, which makes it sufficient as a DR site. “Although there are some areas still under deliberation, we have confirmed that applications can run on Azure environment at the AKS layer,” he says. Although some issues still remain, he stresses that this project should be highly effective from a business continuity perspective.

Microsoft’s support played a major role in driving the project forward, starting with its proposal of SQL Server 2022 at the deliberation stage. “Ever since we participated in EAP, we have received close support, starting with advice on how to initially proceed, along with technical assistance,” says Mr. Hong. He also goes on to note happily that, “As there were almost no case studies and documentation to refer to, I believe that receiving technical assistance from Microsoft headquarters to confirm elements such as configuration was one factor that enabled our project to flow smoothly.” Mr. Yoshikawa also highly values Microsoft’s support, saying, “They offered much support, including creating sample queries.”

As of the end of February 2023, the project is in the overall testing stage while also an environment for peripheral systems is simultaneously being built. From March on, UAT, acceptance testing, and Dry Run testing with an environment that equaled that of the production environment were conducted, and from June and on, a step-by-step release (transition from the existing system) is scheduled, starting from the “savings” service.

“As our company provides a wide array of services, we started with the savings FX service, which has a relatively smaller impact. We will conduct overall tests for FX services with more customers, prepare for acceptance, conduct UAT and make other preparations toward the scheduled release in November. To be honest, from the developer’s side, we would rather release everything all at once. However, when seen from a customer perspective, switching services from those that have less impact on customers is a natural course to take. Yet, because the current system is extremely tightly knit, we had a lot of difficulty separating services to proceed with the project. This experience established simplifying and loosening the system framework an important goal in our next-generation trading system. This time’s project will have individual applications act as containers which will be able to be moved in those units, allowing us to change the number of units to launch according to the number of customers. By establishing a mechanism that does not depend on the environment on which applications are running, I believe we are able to create an environment that allows for small starting steps for ‘test drives on a public cloud’ before investing in the larger infrastructure,” (Mr. Yoshikawa).

Although the overall performance is presently in the process of being checked, Mr. Yoshikawa states that the system is showing solid results from a security perspective, even at this early point in time. “For instance, the existing system had many concerns related to maintaining system confidentiality, as there were cases in which various configuration information, such as database connections, were written down plainly within application configuration files. Our Next-generation Trading System can hold information like this within the foundation of applications secretly on AKS. This allows developers to use the system without being conscious of contents, and also enables easier management of confidential information,” he says.

For personal information stored in databases, Mr. Yoshikawa also notes that SQL Server is able to control what information is visible or invisible according to the permissions the connecting users have. The conventional system operated with a process that would submit queries to mask data and update large amounts of data before being released. In this project, there have been major steps made, not only security-wise, but also toward alleviating operational loads. 

Strengthening connections with Azure, promoting hybrid cloud environments, and striving for multi-clouds

SBI Liquidity Market wants to reflect the results obtained from this initiative to other systems operating within the company. Mr. Yoshikawa says that Microsoft services are the core of their system construction. The company adopted SQL Server 2022 for the database and also AKS, the Kubernetes platform offered by Microsoft, as the foundation for application operation. He also says that they have great expectations for Microsoft for future developments.

“Due to various restrictions related to financial services, we have built AKS upon a Windows Server. As Microsoft offers Azure Stack HCI as HCI solution, we want to utilize these services to reinforce connection with Azure to create a hybrid environment that is both on-premises and in a cloud environment. As I mentioned earlier, we hope to make small starts by first operating on a public cloud before moving to an on-premises environment after the trading volume has increased” (Mr. Yoshikawa).

SBI Liquidity Market plans to develop its business with a multi-cloud method that selects the optimal elements according to purpose and application. Mr. Yoshikawa says that in recent times, mission-critical operations are also conducted on clouds, and he forecasts that the multi-cloud strategy will become important from an availability and business continuity perspective. Furthermore, although the company currently has constructed DWH upon a public cloud that is not Azure, they say that their Next-generation Trading System focuses on close collaboration with Azure, and that they are looking at utilizing Azure Synapse Analytics along with data and AI services offered by Azure.

SBI Liquidity Market has conducted an early deployment of SQL Server 2022, which was officially released in November 2022, enabling the company to boost their FX trading system performance, reinforce security, and ensure availability. Microsoft will continue to support the company.