Mizuho's Finance-Specialized LLM Achieves Passing Level on Banking Practical Exams
Mizuho Financial Group has developed a "finance-specialized LLM (Large Language Model)" trained on banking-specific business knowledge, rules, and procedures, which has achieved passing-level performance on banking practical exams. This model has acquired complex industry-specific rules and regulations unique to the financial sector, as well as internal procedures and business knowledge not typically learned by general AI systems. With this achievement, Mizuho aims to first improve internal operational efficiency and, in the future, dramatically enhance service quality for customers by moving into the development and verification phase for application to more specialized business areas.
Background
In recent years, generative AI technology has evolved rapidly, but general AI models have been insufficient in understanding financial industry-specific terminology, complex regulations, and company-specific business rules, limiting their application to highly specialized business operations. Therefore, Mizuho decided to develop a finance-specialized LLM to directly train AI on the proprietary knowledge and expertise accumulated over many years, which is the source of its competitive advantage.
Overview of Initiative
Using a publicly available open-weight model as a base, the system was intensively trained on vast amounts of internal manuals, training materials, and past approval documents. The model's features and verification results are as follows:
- [Features] Combination of Multiple Learning Methods and Technologies: In addition to continual pre-training using training materials, the approach combined supervised fine-tuning using past exam questions and RAG technology that allows the system to reference relevant information when responding. This enabled the acquisition of banking business-specific knowledge, which is difficult for general AI models to achieve.
- [Results] Passing Performance on Practical Exams: When this model was tested on banking practical exams (subjects: deposits, loans, foreign exchange, financial analysis, etc.), it achieved passing-level performance.
For the learning infrastructure of this development, Amazon Web Services (AWS) was adopted, ensuring that highly confidential data is handled in a secure environment with guaranteed security.
Future Outlook
Building on this achievement, Mizuho will advance the sophistication of finance-specialized LLMs through the following three-stage approach:
- Stage 1 - Finance-Specialized LLM] (Currently Achieved): A model trained extensively on financial domain expertise including fundamental financial knowledge, laws and regulations, and internal procedures, supporting general inquiries and document preparation.
- Stage 2 - Domain-Specific Specialized LLMs] (Next Phase): Training on specialized data specific to each department such as "lending," "legal," and "markets." At this stage, the aim is to realize advanced practical support utilizing knowledge equivalent to veteran bank employees, including procedural guidance based on internal rules (help desk support), credit judgment assistance, and draft creation of approval documents.
- Stage 3 - Collaborative Expert LLMs]: A concept involving the coordination of multiple domain-specific specialized models. For example, when examining corporate support measures, models such as "market analysis-specialized LLM," "legal-specialized LLM," and "credit-specialized LLM" would each analyze from their respective perspectives, and the results would be integrated to propose optimal solutions. This aims to support cross-departmental decision-making that would be difficult for a single expert.
To realize this roadmap, Mizuho will utilize optimal models in appropriate contexts without depending on a single technology. In addition to the model that has achieved the current benchmark, research and development of finance-specialized LLMs continues, with plans to comprehensively evaluate and utilize "answer accuracy," "processing speed," and "cost" for each business operation.
Mizuho is also working to strengthen its proprietary infrastructure in anticipation of handling more highly confidential data and broader internal utilization.
By leveraging the best technologies from diverse options, Mizuho aims to provide more convenient and secure financial services for customers.
Norbert Gehrke
