How to introduce AI technologies into a medium-sized office (May 2024)

Abstract: Here we present a plan to introduce AI technologies into a typical IT office with an overview of technologies available as of May 2024. A wise approach would suggest starting with a limited pilot project first.  

1 Hypothetical scenario

In an era of advancing AI technologies, companies are met with a change in how to introduce AI-enabled technologies and tools into regular work processes. Typical dilemmas companies have are: what technologies are available, what hardware do they need, and where to start.

The question is of course, will AI tools deliver to the promise, i.e. will they be useful as it is promised? Since licenses and hardware are expensive, seems prudent to evaluate tools first.

In this text, we discuss a hypothetical introduction of AI technologies into some medium-sized 100-person office/company, which is focused mainly on IT-related services/development. We are talking from the point of view of a moment in time being May 2024 and news and technologies available. We assume the hypothetical company in question has no previous experience with AI tools but has a curiosity and interest in seeing if such technologies can be useful. The hypothetical office we have in mind is typically based on x86/Windows platform and various development tools.

2 Plan with two directions/focuses

I would divide the plan into 2 directions:

1. The main direction, focused on the Microsoft AI initiative, 85% of the effort
2. The auxiliary direction, focused on alternative technologies, 15% of the effort

2.1 The main direction focused on the Microsoft AI initiative

Microsoft has put a lot of effort into thinking about where and how in the modern office workplace and developing environment AI technologies can be applied to and in which way they can be helpful. It is not just Microsoft because of Microsoft, they did a really good job thinking and planning over a longer period of time. The result is a set of hardware and software products that are planned and produced, not all finished and released at this moment of time.

The main direction was set by Microsoft event on May 20th, 2024, with the proposed CopilotPC hardware standard and Copilot family of products. These are BIG, BIG changes that are setting a path for the future in personal computing. They basically want every PC to have a powerful Neural network processor (NPU) and are writing products for it. I know of no other company at the moment that is so bravely putting a vision for both hardware and software and making ready-to-use AI products in such an impressive portfolio.

So, Microsoft is saying “Stay with us, we have a vision for AI and AI products for you”. And they sound serious to me. What is the catch? Microsoft wants to make money from its hard work and licenses are expensive. For example, for office products, Copilot for Microsoft 365 is $30 per user per month. For an office of 100 employees, that is $36,000 per year just for software licenses to have smart AI-enabled Office products. If you want AI-enabled other products, you pay for separate licenses again. And is no longer you buy a product and use it forever, the new trick is they sell a license “per user per month”.

Still, because of their clear vision and a big portfolio of products, if you want to go AI direction, the Microsoft CopilotPC initiative is the way to go.

But, stay grounded regarding expectations of AI-enabled products. Try before you buy, because marketing is very strong and they tend to overpromise on the capabilities of AI-enabled tools. Even supercomputer-based ChatGPT-40 was found to “hallucinate” 3% of the time, so is not really “production ready” and requires output to be checked/proofread by the humans. Be careful, many AI companies, including Microsoft, just want your money and will sell you anything under the marketing name AI-enabled.

Also, most of the Copilot and other AI-enabled Microsoft products are still in development and new to the market, so maybe it is wise to wait a bit for them to mature and get bugs-free. Taking early versions, you might find yourself in the position of practically real-world testing and discovering of and reporting of unknown bugs.

The introduction of Microsoft AI technologies can be divided into 3 elements:

1. Acquiring CopilotPC hardware

2. Acquiring AI-enabled software. That software can be divided into 2 groups:

   1. 2A) Software focused on AI-Cloud services. It can be introduced without introducing CopilotPC hardware first.
   2. 2B) Software focused on CopilotPC hardware. You need to buy CopilotPC hardware first to run this software.

3. Acquiring training tools: videos and books/manuals (PDFs)

2.2 The auxiliary direction, focused on alternative technologies

Here I would put an effort into seeing/monitoring the usability of other technologies, at the moment of writing, I find personally interesting technologies:

1. ChatGPT-4o by OpenAI
2. Meta Llama 3 OpenSource AI

The reason I put ChatGPT-4o by OpenAI  into the auxiliary group is, that while the tool is very entertaining, I do not see it very easily integrated into the work process and other tools. Maybe I am wrong on this one, but if I need to generate an email in ChatGPT and then copy-paste it every time, it is not going to be very efficient. That is why Microsoft Copilot tools with integration are supposed to be much more efficient. Time will tell us what is the best.

I work in an IT-developing environment, and professionally made, license-free, tools like Meta Llama 3 Open Source AI are always interesting. Microsoft is planning on providing Small Language Model (SML) Phi Silica, specifically designed for new CopilotPC hardware. That way developers will have the ability to develop custom AI systems on CopilotPC platform. Currently, Meta Llama 3 requires powerful Linux hardware but is license-free, and let’s just keep an eye on it.

3 Buying new CopilotPC compatible hardware

In its vision, Microsoft is proposing a new hardware platform: CopilotPC. There are 2 versions, CopilotPC/x86, and CopilotPC/ARM.

• CopilotPC/ARM hardware/laptops will be available as of June 2024.
• CopilotPC/x86 hardware/laptops will be available later. With Intel chips expected to be finished in Q3 2024, laptops will be available probably December 2024.

3.1 What is your local IT department saying

If you are eager to catch the AI wave and have a strong local IT department, go for CopilotPC/ARM right now. Windows 11 is compatible with that platform, as are many Microsoft applications. Adobe is in the process of migrating its products to CopilotPC/ARM/Windows.

I work in a 100+ people office that is mostly x86/Windows based and the IT department is quite busy supporting and managing all the user's problems and their hardware/laptops.

Typical IT-department consists of engineers of various experience and skill levels. Employees tend to fluctuate, so you can not base your enterprise technology base on one person. Typically, IT engineers have good experience with x86-64 hardware platforms and relevant hardware and admin tools. If you push them to a new ARM laptop platform, many will feel dislike in their work. Even worse, despite claimed compatibilities, some of their tools might not work on new hardware. The problem can be serious and cause big delays if for example their favorite USB-booting tool is no longer working on ARM-laptops platform, or if the licensed disk-imaging tools they used for years do not work in the new environment.

If you can wait until December 2024 with AI-technologies introduction, maybe is worth playing safe and wait for availability of CopilotPC/x86 hardware. You will make life much easier for your local IT department and all x86 tools will still work. You can still in meantime introduce those AI tools that require only AI-Cloud services.

4 Used AI-technology company levels

You need to define to which level you want AI technology introduced into the company. I see several levels/objectives that can be reached with the available tools and services on the market:

Objecitve1) Usage of commercial productivity AI tools in regular office work. So, depending on your office/company business domain, you want to start using AI-enabled tools like Microsoft Office 365 Copilot, or some of the new AI-enabled Adobe productivity tools. Again, they can be divided into 2 groups, those that are exclusively based on AI-Cloud services and those that require CopilotPC hardware, like most new versions of Adobe products.

Objective2) Usage of AI tools for software development. If you are a software developer shop, you will definitely be thrilled with the idea of enhancing productivity with AI tools. Here I talk about using tools like Microsoft GitHub Copilot or ChatGPT to generate snippets of code.

Objective3) Creating AI-enabled applications. Sooner or later, developers will start producing AI-enabled applications, and for that they need some tools and smarter hardware. Microsoft has announced the availability of Phi-Silica, which is a Small Language Model (SML), specifically designed for the Neural Processing Units (NPUs) in new Copilot+PCs. Also, Meta Llama3 is an Open Source of Large Language Model LLM for the Linux platform.

4 Start with Pilot projects

It is always good to start with Pilot projects. That will put less stress on IT department if they need to prepare new hardware for a limited number of users first and gain the necessary experience with installing and maintain the new platforms like CopilotPC/x86/Windows.

IT department also needs to get some experience with Copilot products and licenses. Also, licenses are expensive and you want to see if the product's AI-enhancements are useful or maybe we need to wait a couple of years until they mature.

I would divide Pilot projects into 2 categories:

1. Pilot-AI-Cloud. Here I mean testing AI-tools that are purely AI-Cloud services based.
2. Pilot-AI-CopilotPC. Here I mean testing tools that require CopilotPC hardware.

4.1 Pilot-AI-Cloud

In this pilot project, I mean testing AI-tools that are purely AI-Cloud services based and you need no particular hardware for them. You can start this pilot project right away. Depending on your business domain, you can test AI-tools of your interest. The key thing is to use new tools in real work process, to get feedback how useful they are in the real world. Some ideas might be:

• Get 2 licenses for GitHub Copilot (different versions are $10, $19, $39 per month) and give them to your Senior Software Engineers to test them in their everyday work. After 2 months you will have in-house experience with AI-tools and an opinion about the usefulness of such tools.
• Get 2 licenses for ChatCPT-4o Plus ($20 per month) and give them to two employees, at least one Senior Software Engineer. The problem is only that Chat GPT Desktop app for Windows users will be available later this year. After 2 months you will have in-house experience are they useful in generating code and in other office work.
• Get 2 licenses for Microsoft Copilot for Microsoft 365 ($30 per month per user). Give them to 2 employees and you will get their opinion on how useful they are.

4.2 Pilot-AI-CopilotPC

In this pilot project, I mean testing tools that require CopilotPC hardware. Some ideas might be:

• You buy 2 laptops, be it Copilot/ARM/Windows or CopilotPC/x86/Windows with licenses, and give them to 2 Senior Employees. You need to migrate completely their work environment to the new laptops, so that AI-tools get used in the real world situations. I see no point in just “playing” with new laptops. They need to be used in day-to-day work.
• You get some CopilotPC-enabled software like Adobe products to test them in real work scenarios.
• You get some CopilotPC-enabled software like Microsoft products (I am not sure which is not Cloud-based at the time of writing) and test them in real-world scenarios.

5 Details on Copilot+PC Microsoft initiative (May 2024)

Copilot+PC is Microsoft's vision related to AI on PC platform, as of May 2024

Here is the summary:

• The main idea is to have AI not only on Cloud but also to bring AI to the PC platform
• Announcement (dated October 2023) envisions new “AI PC” hardware. Shortly, the new architecture proposes 3 processors in PC, Main CPU, Graphical processor (GPU), and a processor for Neural Networks (NPU- Neural Processing Unit). They talk about 100 million new AI-PCs in 2025. Intel shipped its first “AI PC”s in December 2023, based on “Meteor Lake CPUs”.
• In announcement (May 20th 2024) Microsoft gives its vision of “Copilot+PC” platform. Basically, they are talking about PC hardware platform that will be able to execute Microsoft AI software. Microsoft sets hardware requirements even higher than previously mentioned AI-PCs. Microsoft standard “Copilot+PC” requires NPU of at least 40 TOPS (=40x10^12 operations per second).
• The main idea is, Copilot+ PCs will be able to handle some AI-accelerated workloads like chatbots and image generation locally instead of relying on the cloud.
• Microsoft office 365 (Word, Excel, PowerPoint, Outlook, Teams) will be Copilot enabled. Copilot for Microsoft 365 provides real-time intelligent assistance, enabling users to enhance their creativity, productivity, and skills.

Copilot+PC hardware comes in 2 flavors:

• ARM-based architecture. Right now, requirements can only be met by a single chip in the Windows PC ecosystem, Qualcomm's Snapdragon X Elite and X Plus. A number of Laptops from Microsoft, Lenovo, HP, Dell, Acer, Samsung, Asus will be available in June 2024. It is claimed that the performance of PCs based on processor Qualcomm's Snapdragon X Elite and X Plus from the list above, is superior to MacBook Air M3.
• X86-64 (Intel/AMD) architecture. At this very moment, none of the Intel/AMD chips can meet the Copilot+PC requirement of 40TOPS. Intel's Meteor Lake-based Core Ultra NPUs get at 10 TOPS, and some AMD's Ryzen 7000 and Ryzen 8000 processors with NPUs get between 12-16 TOPS. A new generation of processors will be released soon: Intel's next-generation processors, codenamed Lunar Lake, will be released in Q3 2024. They are expected to have 45+ TOPS NPU, with details of performance published later. Intel expects to sell "more than 40 million" AI PCs in 2024.

6 Details on Microsoft's new brand “Copilot” (As of May 2024)

There have been recently a number of IT articles using the word ‘Copilot” in different contexts. I was also puzzled for a while, what do they exactly mean when they mention “Copilot” in different texts? Over time it started to make some sense.

6.1 What does the name Copilot mean?

I will try to provide my understanding, based on articles I see and also based on the way it is used in practice, at the present point in time, May 2024. Brief explanations in plain language are:

• Microsoft uses the brand name Copilot to refer to its own products, services, or technologies that are somehow related to AI
• Name or addition of name Copilot means that product or service is “AI-enabled”, without specific criteria what kind of AI is used or the power of AI enhancement. That estimate of what is AI-enabled or not is at discretion of the Microsoft. It can be enhancement from some simple “smart heuristic algorithm run locally” to a “full remote call to a cloud-based supercomputer”.
• Copilot can refer to software or hardware technologies alike
• Copilot can refer to AI technology executed locally on PC or AI technology executed remotely on the cloud, or a combination of both. From the product name like “Copilot Xyz” can not be concluded if they mean local or cloud execution of AI services that enhance product Xyz.
• Copilot service/application can mean some algorithm run on plain c86 PC under Windows 10, or some service run on Windows 11 on PC with neural processor NPU. They use the brand name Copilot for both, for applications run on regular PCs and applications run on PCs with NPU.
• Proclaimed principle is kind of: 1) Run short AI tasks locally on PC with NPU, because of small latency. 2) And run big AI tasks on the cloud, because of more processing power, with bigger latency in communication.
• “Copilot+PC” is a hardware standard by Microsoft
• Name Copilot originates from the Microsoft Copilot chatbot, but now is used with a different, broader meaning for all AI-related technologies
• It seems that Microsoft itself is not sure where this AI story is going, so they do not have a clear policy on what is under the umbrella Copilot and what is not. So, looks like marketing kicks in, and for better sales they just add Copilot to the product name if they think that will make some money.
• “Copilot Pro” is a subscription to AI cloud services for $20 per month. The way I see it, they are partners with OpenAI, so that means your cloud request will be basically piped to ChatGPT-4o server. It is kind of vague, but if you buy a product “Xyz Copilot”, you pay for a license more than just for the product Xyz. What you get is local AI services executed on your PC. And maybe some (or none) AI services on the cloud with limitations. And the subscription Copilot Pro gives you access to more services on the cloud for support of the application Xyz. That means bigger bandwidth, lower latency, more requests, more documents/images processed, etc.
• So, typically you have the product “Xyz”, and you pay a license for it. Then you have AI extension called “Copilot Xyz”, and you pay a license for it which might include some limited access to AI-Cloud. Then you optionally might pay a license for “Copilot Plus” to have better/faster access to the AI-Cloud.

6.2 Copilot Products – AI Products by Microsoft

Most of the Microsoft AI products include the brand name Copilot, but not necessary. Sometimes they use other names, I would guess for marketing reasons. Here is a list of some frequently mentioned Microsoft AI products (as of May 2024) with one-liner descriptions.

• Copilot for Microsoft 365. That is integration of the Copilot AI assistant into Microsoft's 365 office family, including Teams, Word, Outlook, Excel, and PowerPoint.
• Copilot in Windows. That is AI assistant in Windows, for example for PC settings
• Microsoft Security Copilot. That is AI enhancement for security, like recognizing cyberattacks.
• Microsoft Copilot for Service. That is AI enhancement for customer relationship management (CRM) applications.
• Copilot for Sales. An AI assistant for sellers and customer interaction.
• Microsoft 365 Chat. The chat-based AI assistant for Office products.
• Team Copilot. An AI assistant for Microsoft's Teams, Loop, Planner, and Microsoft 365 applications, that acts as a manager/secretary for the team. It has the roles of Meeting Facilitator, Group Collaborator, and Project Manager.
• Copilot Studio. This is a no-code or low-code interface for creating custom AI assistant based on for example HR documents.
• Recall. That is AI-enabled spying software, like a smart keylogger, that keeps a history of all activities on the computer.
• Cocreator. That is a generative AI application that creates pictures based on text prompts or sketch input.
• Live Captions. Provides captions for any audio in 40 languages.
• Windows Studio Effects. The AI assistant can help you manipulate videos on your PC.
• GitHub Copilot. The AI coding assistant for Visual Studio and VS Code.

Based on different articles, some products are available only on Windows 11, and some only on “Copilot+PC” hardware. Also, at the time of writing (May 2024) not all products have been finished and released.

6.3 AI Licenses are expensive

Microsoft wants to make money from Copilot products/AI products since it invested significant money and resources into the AI software revolution. Licenses are quite expensive Here are some examples:

• Copilot Plus - $20 per month per person
• Copilot for Microsoft 365 - $30 per user per month
• Copilot in Windows 11 – free with Windows 11
• Microsoft Copilot for Service - $50 per user
• Copilot for Sales - $50 per user
• GitHub Copilot - $19/$39 per user per month

7 Details on Adobe products (As of May 2024)

Adobe is in the process of migrating its product family to CopilotPC/ARM/Windows platform.

Photoshop, Lightroom, Adobe Express, Adobe Firefly, and Adobe Acrobat have been already migrated and Premiere Pro, After Effects, and Illustrator are coming soon.

That will give users a better experience with the latest AI tools like Generative Fill in Photoshop and Generative Remove in Lightroom.

One can expect that in the future Adobe will stop making versions of its products for “legacy PC”, and will target only CopilotPC platform.

That will increase the popularity of CopilotPC/ARM/Windows platform for ordinary users. No one will be interested anymore in buying a "legacy PC", which is not CopilotPC compatible. I believe that in 2 years all new PCs will be CopilotPC compatible, be it CopilotPC/ARM or CopilotPC/x86-64.

8 Details on ChatGpt-4o (As of May 2024)

8.1 AI app ChatGpt-4o is a huge success with mobile phone users

Although ChatGpt-4o initially when installed comes as a freeware, the application is hugely popular with mobile phone users who are buying subscriptions of $20 per month to get faster access with more bandwidth/options.

Based on available data daily revenue from the app in May was around $900,000, and that accounts for OpenAI first paying Google and Apple for using their app stores.

This is proof that AI applications are of interest to ordinary users.

8.2 Voice input is key to success

What is fundamentally different and new in this version of ChatGpt is the ability to communicate in natural language via voice with AI-Cloud service. App is fluent in 50 languages so it attracts users from all over the world.

It seems many ordinary users are repelled from the usage of AI services if they need to type a lot.

8.3 ChatGPT desktop app is available for Mac, but not for Windows yet

ChatGPT desktop app is available for Mac for “ChatGpt Plus” subscribers. Desktop app for Windows users will be available later this year.

9 Details on Microsoft Phi Silica (As of May 2024)

Microsoft introduced Phi-Silica, a 3.3B parameter AI model optimized to run on Copilot + PCs NPUs. The target audience is Windows developers developing applications in Copilot+PC environment.

In order to enable Windows developers to exploit new hardware platforms, Microsoft introduced Phi-Silica, a 3.3B parameter model optimized to run on Copilot + PCs NPUs.

Phi-Silica is a Small Language Model (SML), specifically designed for the Neural Processing Units (NPUs) in new Copilot+ PCs. Phi Silica is the smallest model in the Phi family SLM models, with 3.3 billion parameters. Phi-3-Silica runs directly on the Copilot+ PC without the need for an internet connection.

Developers can access Phi-Silica using Windows App SDK, and some C# examples are available on the Microsoft site. Phi-Silica can be used to develop applications for the Windows ecosystem.

Typical use cases are expected to be: Offline voice assistants with limited functionality; Text-to-speech conversion for visually impaired users; Real-time captioning for audio and video; Real-time language translation; Offline comprehension; Smart dictation and speech recognition.

10 Details on Meta Llama 3 (As of May 2024)

Llama 3 (acronym for Large Language Model Meta AI) is a family of LLM released by Meta AI in April 2024. Llama 3 is open source. Meta released Llama-3 with two sizes: 8 billion and 70 billion parameters. Hardware platforms from AMD, AWS, Dell, Intel, NVIDIA and Qualcomm support Llama 3. Llama 3 is free as long as it is used under the terms of the license.

It is purely my speculation that with advancements in CopilotPC hardware with more powerful NPUs, Meta might tailor a version of Llama for platforms like CopilotPC/ARM/Windows or CopilotPC/x86/Linux.

11 Conclusion

The presented plan is the opinion of one software engineer and AI enthusiast and is more kind of a set of ideas to inspire and provide some advice to anyone interested. Each company/office is specific and will have its own take on the problem. Take anything you like from this plan, and add your own ideas, to create your company’s custom plan.

12 Notes

• The official name of the Microsoft hardware standard is “Copilot+PC”, but many authors shorten it to CopilotPC
• Many of the above suggestions are based on very recent news, Meta Llama 3 was released on April 18th, and CopilotPC standard on May 20th. At the time of writing (May 31st, 2024) not all of the above-mentioned products are finished nor released. Neither are the full lists of specifications available.