The costs of operating an AI platform

Introduction

Artificial Intelligence (AI) is rapidly becoming a part of our daily lives. From virtual assistants to recommendation systems, AI is making a significant impact on how we live and work. However, building and operating an AI model can be expensive. In this blog post, we will discuss the costs associated with operating an AI model and how to manage them.

Infrastructure Costs

One of the most significant costs associated with operating an AI model is the infrastructure required to run it. AI models require large amounts of computational power and storage. This means that you will need to invest in high-performance hardware such as GPUs and CPUs. Additionally, you will need to provide a robust and reliable network infrastructure to support the communication between the various components of the AI system. All of these requirements can quickly add up, making it a significant cost for small and medium-sized businesses.

Using ChatGPT to write cover letters, generate lesson plans, and redo your dating profile could cost OpenAI up to $700,000 a day because of the pricey tech infrastructure the AI runs on, Dylan Patel, chief analyst at semiconductor research firm SemiAnalysis, told The Information.

In a phone call with Insider, Patel said it's likely even more costly to operate now, as his initial estimate is based on OpenAI's GPT-3 model. GPT-4 — the company's latest model — would be even more expensive to run, he told Insider.

Companies using OpenAI's language models have been paying steep prices for years. Nick Walton, the CEO of Latitude, a startup behind an AI dungeon game that uses prompts to generate storylines, said that running the model — along with payments to Amazon Web Services servers — cost the company $200,000 a month for the AI to answer millions of user queries in 2021, CNBC reported.

Data & Training Costs

Another significant cost associated with operating an AI model is the cost of data. AI models require large amounts of data to train and fine-tune the algorithms. This means that you will need to invest in acquiring and processing data. Acquiring data can be expensive, especially if you need to purchase it from third-party providers. Furthermore, data processing can be time-consuming and requires specialized skills, which can add to the overall costs of operating an AI model.

Tirias Research forecasts that on the current course, generative AI data center server infrastructure plus operating costs will exceed $76 billion by 2028, with growth challenging the business models and profitability of emergent services such as search, content creation, and business automation incorporating GenAI.

Analysts and technologists estimate that the critical process of training a large language model such as GPT-3 could cost over $4 million

Meta’s largest LLaMA model released last month, for example, used 2,048 Nvidia A100 GPUs to train on 1.4 trillion tokens (750 words is about 1,000 tokens), taking about 21 days, the company said when it released the model last month.

It took about 1 million GPU hours to train. With dedicated prices from AWS, that would cost over $2.4 million. And at 65 billion parameters, it’s smaller than the current GPT models at OpenAI, like ChatGPT-3, which has 175 billion parameters.

Maintenance and Support Costs

Once you have built and deployed an AI model, you will need to maintain and support it. This includes updating the software and hardware, monitoring the performance of the system, and fixing any issues that arise. Maintaining an AI system can be challenging and requires specialized skills, which can be expensive to hire. Additionally, you may need to provide ongoing training and support to the end-users of the system, which can add to the overall costs of operating an AI model.

To use a trained machine learning model to make predictions or generate text, engineers use the model in a process called “inference,” which can be much more expensive than training because it might need to run millions of times for a popular product.

For a product as popular as ChatGPT — which investment firm UBS estimates to have reached 100 million monthly active users in January — Curran believes that it could have cost OpenAI $40 million to process the millions of prompts people fed into the software that month.

Conclusion

Operating an AI model can be costly. From infrastructure and data costs to maintenance and support, there are many expenses associated with building and operating an AI system. However, by carefully managing these costs, you can build a successful AI system that delivers value to your business. By investing in the right infrastructure, acquiring and processing data efficiently, and providing ongoing maintenance and support, you can minimize the costs associated with operating an AI model and maximize its potential for your business.

‍

‍