Artificial intelligence is one of the hottest topics of the moment for both tech companies and developers. With the increasing availability of data, we are witnessing a wider adoption of machine learning in software applications. However, this spread brings with it new requirements, most of which need a reliable cloud infrastructure to support the high computational needs of analyzing and using data.
This article outlines these new demands, and describes IBM‘s approach to the journey towards the adoption of AI solutions: the AI Ladder.
Data, Cloud and AI: the AI Ladder
We are currently witness to the crucial role of digital transformation as it impacts on every industry and business. In this context, the high availability of data, alongside an increasing number of AI and machine learning-based applications, plays a prominent role.
Besides data and AI, there is also a third factor that can significantly ease the digital transformation process – cloud computing. Using IBM employee Hemanth Manda’s interesting metaphor, it’s possible to see data as the fuel, Cloud as the vehicle, and AI as the final destination. All of these components are required to proceed along the path towards digital transformation.
To be successful in this journey, companies need to modernize their solutions, revise workflows for data preparation and improve the resulting AI application workloads. This entire process requires a shift towards cloud technologies.
Despite the evident importance of embracing cloud technologies, many companies remain reluctant to immediately embrace the public cloud. This reluctance is not usually related to the benefits that cloud architecture offers, but are often due instead to regulatory concerns about operating on a public cloud (consider European GDPR rules, for example). Consequently, they prefer to adopt private cloud solutions in order to have greater control of their data and to comply more easily with local laws.
The perfect answer to this conundrum seems to be the adoption of hybrid cloud solutions, especially for companies that are transitioning to a cloud architecture in order to modernize their workflow. Hybrid cloud solutions combine a private cloud with one or more public cloud services, often relying on ad-hoc software to allow communication between them.
But why are we talking about the Cloud, when we started our discussion with AI? The answer is not straightforward, but is summarized by the following sentence:
“There is no AI without IA (Information Architecture)”
This is where the cloud and the AI Ladder come to our aid.
What is the AI Ladder
According to a recent report on the adoption of AI within companies, 81% of business leaders do not understand the data infrastructure required for their own AI solutions. There is no way you can properly optimize performance, or even improve the maintainability of AI solutions when a poorly designed data architecture is in place.
On the other hand, defining a solid data infrastructure is no easy task, requiring a deep knowledge of data and the AI solutions developed on top of them, which may sometimes be unknown at the time of design. It is therefore no surprise that IBM has defined a prescriptive approach to help customers overcome these kinds of challenges, and speed up the whole digital transformation process. This approach is known as the AI Ladder.
The AI Ladder allows companies to simplify and automate the process of converting data into useful information. It consists of four different steps (often referred to as the ‘rungs’ of the ladder), namely:
- Collect: this first rung aims to make data simple and accessible
- Organize: here the data are processed to create a business-ready analytics foundation
- Analyze: at this stage, data are ready to be effectively analyzed, so the goal is to build and scale AI with trust and explainability
- Infuse: after data analysis, the goal of the final rung is to operationalize AI throughout the business
Off the ladder, although spanning all its rungs, is the Modernize concept. This represents the process of simplifying and automating data preparation and processing data within a multi-cloud platform.
Modernize
Before taking a first step onto the AI Ladder, it is best to start by looking at how to modernize your data architecture. In this context, ‘Modernize’ means building a proper information architecture for AI, with the aim of gaining the flexibility necessary to meet today’s demands and remain competitive in the future.
As indicated previously, a reliable agile architecture can be built by opting for a hybrid multi-cloud platform. Such solutions allow organizations to use their data and applications across public or private Clouds, by exploiting the power of containers. With such an approach, data management can be carried out effectively, even when extreme data proliferation spreads information across several databases and clouds. On the other hand, if an organization deploys and runs AI projects in specific cloud environments, they are forced to use the tools provided by that particular platform, with consequent limitations on what can be achieved.
Finally, given the dynamic nature of AI, it is crucial for organizations to automate AI life cycles through collaborative and agile workflows. In this way, data can be made ready for any AI application, open source can be put to work, and AI infused across teams.
Collect
Once the foundation of our modern data architecture is solid, we are ready to explore our data. However, the information is frequently ‘hidden’ in some way, making it difficult to use data as they are. This is due to the variegated nature of the sources we use for data, a phenomenon often referred to as siloed data, i.e. data that is difficult to access from other systems, which makes gaining valid and significant analytical insights impossible.
Consequently, the first rung of the AI Ladder should consist of the first step towards data preparation – making our source something that is simple and accessible. There are many examples available to give you an idea of how a company can address such issues.
The first option to consider might be to store all data types. Imagine a company that stores only historical transactional data; while AI solutions could clearly work on such data, it wouldn’t be possible to access other kinds of data that might be useful in the future. Collecting more data types can ease AI processes, leading to more effective solutions, now and in the future.
Another option involves investing in AI-infused data management tools. Instead of relying on traditional data storage repositories, an AI-powered platform offers a broader suite of capabilities, which will perform better as data grows.
The ‘Collect’ rung is not just about data quality, but also relates to the ability to access more data, resulting in the fueling of smarter AI applications.
Organize
Having improved our data collection processes, are we ready to use them? In many cases, companies are able to collect a huge amount of various kinds of data, but end up not knowing what they actually have in their storage systems, which processes use their data, and how to remain compliant with laws and regulations.
This rung of the AI Ladder aims to organize data properly – a process that involves solving three key issues:
- Address data quality
- Organize and catalog data
- Govern data
Address data quality
In order to organize data properly, it is necessary that they are ‘business ready’, which means that they need to be cleansed, and verified as not incomplete, i.e., ready to be used to build an AI model. When data are not business ready, their use can significantly impact the productivity of several stakeholders within the company, including data scientists, data analysts and even end users.
If you think that this process can be easily carried out, consider that organizations spend 80% of their time just preparing data to make them business ready.
Organize and catalog data
Once the data quality is sufficiently good, it is essential to organize them properly. Think about a library – without a proper catalog you wouldn’t be able to find a single book. The analogy is quite straightforward: you need a catalog of the data you are storing, including information about the source, the owner(s), metadata and anything else you think might be useful to better identify your data.
Govern data
Finally, an organization must govern their data; it must ensure that they are only accessible to those who have the proper permission level. Securing data is crucial nowadays, and not just because of practical consequences (think of the potential outcomes if some bank data were to leak), but also because doing so is fundamental to remaining in compliance with current legislation around the world (e.g. GDPR in Europe).
Analyze
At this stage, the organization has collected its data and organized them properly. Now it’s time to build the actual AI models and scale them across the business. This idea has one key goal: to gather insights about how the models work, and be able to manage them with transparency. To this end, it is important to follow the AI life cycle, which includes the following stages:
- Build: the company creates the AI model. At this stage, it is crucial to choose the proper algorithms and evaluate their performance in order to select the best parameters
- Run: once the model is built, it is time to put it into production, often within a more complex application workflow. At this stage it is critical to let the model make the proper decision and/or predictions
- Manage:when the model has been built and run, it is time to investigate its behavior in more depth. The huge amount of data may make it difficult to identify dark patterns, which could prevent the model from behaving properly in certain situations. It is therefore crucial for companies to be able to explain how their AI applications arrive at a specific output, thereby eventually identifying any unexpected actions that may occur. This avoids any possible bias, and makes actions both explainable and transparent
Establishing trust and transparency is essential at every stage of the AI life cycle.
Infuse
The final step on the AI Ladder is about applying the AI models across your enterprise’s processes. We all know how large the number of applications where AI models can be applied is, from customer care (think of chatbots and virtual assistants) to medical applications (see this article in our magazine, for instance), and in many other fields including marketing, HR, supply chain, and so on. In some cases, applying AI to existing workflows may require organizations to revise these significantly, or develop new workflows, perhaps across multiple departments.
Such processes may appear complex; however, they represent the last step towards a modern, effective, AI-infused workflow, which can, in turn, significantly improve the whole company’s performance.
The AI Ladder: main takeaways
It should be clear from the discussion above how important it is to modernize company data infrastructures, especially considering the easy availability of hybrid multi-cloud platforms. Remember, “there is no AI without IA”, but don’t lose sight of the fact that AI is not magic: it needs hard work, the proper tools and the right methodologies. The IBM AI Ladder represents a valuable paradigm to help companies in following the right path towards the modernization of their infrastructure.
