ChatGPT, a chatbot launched by OpenAI in November 2022, has quickly garnered attention for its detailed responses and articulate answers across many knowledge domains.
How much does ChatGPT understand a particular domain like BIAN? BIAN (Banking Industry Architecture Network) is a consortium of banks, technology providers, and consultants that work together to establish a global, open standards framework for the banking industry. BIAN has made available much of its work to the public. However, some contents, including work in progress by various BIAN working groups, are only available to members through the member-only Wiki. Some learnings and knowledge are likely only known to a small number of people working together more closely over the years.
I devised a set of questions to test ChatGPT’s understanding of BIAN. I started with some basic factual-based questions to questions on more advanced topics. Then I dived deep into the BIAN adoption topic, where patterns and best practices are emerging. Finally, I asked questions regarding the ongoing work that is likely not well-known in the public domain yet.
Basic factual-based to more advanced questions
Question #1: Please tell me a bit about BIAN.
The first time, ChatGPT answered: BIAN stands for “Blockchain Interoperability Alliance Network”, a not-for-profit organization that focuses on developing and promoting open standards for blockchain interoperability…
After I told it that by BIAN, I meant Banking Industry Architecture Network, ChatGPT could learn and use that in future chat sessions. Is the learning carried out system-wide? Please let me know the answer you get.
ChatGPT did not get all the facts right — BIAN was founded in 2008 and is not headquartered in Switzerland. It described things with equal authority, whether wrong or right. If it is not reviewed by an expert to identify any such errors, it could quickly propagate massive amounts of disinformation that could generate its own echo chamber of nonsense. [Thank you, Guy Rackham, for this excellent point.]
Question #2: What are the design principles of BIAN?
Good answer. ChatGPT listed eight relevant design principles: interoperability, openness, modularity, reusability, etc.
However, it missed a very important one. Here is the comment from Guy Rackham, the BIAN lead architect: My biggest beef — the key behind BIAN, its ‘secret sauce’ is that it employs a very specific approach (“commercial asset leverage”) to identify truly canonical conceptual building blocks — that are behavioral in nature. This is something that you need to understand to fully leverage the model, and something that very few people are aware of. It is also something that is not well documented — and so it was completely overlooked by ChatGBT. This demonstrates the limitation of a tool that basically reworks existing material based on some kind of semantic algorithm. It does not apply any intuition or interpretation, but presents its output in a form where the unexpecting may assume it is complete and correct.
Question #3: Can a Control Record have multiple instances?
Good answer.
Question #4: Does a Control Record have structure, or is it divisible?
Not a satisfactory answer. I would expect it to introduce Behavior Qualifiers that break down the work performed by Service Domains as captured in their Control Records, and the “fractal” structure.
Question #5: Why do not you tell me about Behavior Qualifier?
Again, not a precise definition of Behavior Qualifier. ChatGPT is devising an answer without knowing/understanding the BIAN definition of the term.
Question #6: What does it mean that Service Domains are canonical?
The answer did not use the language from the BIAN document. In BIAN, Service Domains are canonical means that their role/purpose is to be consistently interpreted from one deployment to another, supporting high levels of interoperability. I feel that ChatGPT combined the meaning of canonical and Service Domains to come up with the answer.
Adoption topics
Using BIAN in implementation often involves elaborating BIAN conceptual models to define the Logical Layer for a specific bank and the Physical Layer with particular technology and infrastructure.
Question #7: Is there a Logical Layer for BIAN?
BIAN provides descriptive conceptual models. The Logical Layer often means the more elaborated design models for a specific deployment. ChatGPT provided its interpretation of the Logical Layer here.
Question #8: By Logical Layer, I meant the models designed for a specific bank.
ChatGPT adapted its answer to my question!
Question #9: Does BIAN provide the Physical Layer for implementation?
Good answer.
Question #10: Is BIAN designed for implementation using different technology?
Good answer.
Question #11: If that is the case, how to develop the Physical Layer for a specific bank based on the Logical Layer?
Very interesting answer. ChatGPT came up with a process for developing the Physical Layer, likely based on something other than the BIAN literature but on its understanding of the general development process.
Question #12: How can an organization adopt BIAN effectively?
Again, very interesting answer. ChatGPT came up with an adoption process, likely based on something other than the BIAN literature but on its understanding of the general adoption process.
Unknown or subjective topics
Question #13: Alright. Tell me a bit about the BIAN EDA working group.
Wrong answer. ChatGPT interpreted EDA as Enterprise Data Architecture, which does not exist as a BIAN working group. Also, the BIAN EDA (Event Driven Architecture) working group has yet to make available its work to the public. ChatGPT should acknowledge it does not know about EDA and try to clarify with the user.
Question #14: That is not quite right. By EDA, I meant Event Driven Architecture.
ChatGPT devised the answer based on the common understanding of EDA and BIAN rather than on the work performed by the BIAN EDA working group.
Question #15: Is BIAN well received by its members and the banking industry?
I know the answer can be subjective. ChatGPT gave an answer that is logical and hard to argue against without facts backing its response. Well, how much can we expect from a chatbot of this nature?
Summary
ChatGPT gave good answers to most of the general factual-based questions that can be inferred from the publicly available information about BIAN. For more complex questions where domain knowledge does not quite exist or may not be available in the public domain, it devised answers based on understanding the elements of the questions. It adapted its responses based on implied relationships in subsequent questions. To enhance the quality of answers to BIAN-related questions, the BIAN community should strive to put more relevant information into the public domain and provide feedback on responses to BIAN-related queries.
I would love to hear your thoughts on ChatGPT’s understanding of a specific domain like BIAN.
Acknowledgment
I want to thank Guy Rackham, the BIAN lead architect, for his timely review of the article and many excellent points, some of which I incorporated into the article.
