How to Become a Cloud & AI Infrastructure Architect: 5 Projects That Can Prepare You for the AI Computing Boom

Everyone wants to build AI applications. Far fewer people understand what happens after the model is built.

Where does a 70-billion-parameter model actually run? How do thousands of users send requests to it simultaneously? What happens when one GPU becomes overloaded? How does a company add more GPUs automatically when traffic suddenly increases? How do engineers know when inference is becoming slow? And how do you avoid paying for expensive GPUs at 3 a.m. when nobody is using them?

Welcome to AI infrastructure engineering. As artificial intelligence moves from experiments into production systems, companies increasingly need engineers who understand the layer connecting models, GPUs, containers, Kubernetes, networking, storage, observability and cloud infrastructure.

For African engineers, this could become an increasingly valuable skill over the next decade. Recent infrastructure deals are beginning to expand the compute available on the continent. Nigeria's UduTech, for example, recently partnered with South Korea's Baro AI to expand access to GPU infrastructure through its Africa GPU Hub initiative. Afritech Connect examined the important distinction between announced hardware and compute developers can actually rent in UduTech and Baro AI Expand GPU Access in Africa — But Can Developers Actually Rent Them?.

Meanwhile, McKinsey estimates African data-centre demand could grow from about 0.4GW to between 1.5GW and 2.2GW by 2030, potentially requiring $10 billion to $20 billion in investment in data-centre shells alone.

Somebody has to design, deploy and operate the software infrastructure running on top of that compute.

That is where AI infrastructure engineers come in.

What Does an AI Infrastructure Engineer Actually Do?

Think about ChatGPT. You type a question and receive an answer seconds later. Behind that simple interface is considerably more than an AI model.

Servers need GPUs. Models need to be loaded into GPU memory. Requests need routing. Containers need deployment. Traffic needs load balancing. Failed workloads need restarting. GPU utilisation needs monitoring. Logs and metrics need collecting. Infrastructure needs scaling when millions of requests arrive.

Engineers also have to control cost. If ten GPUs are running when only two are required, the company is burning money.
AI infrastructure engineering is essentially the discipline of making AI reliable, scalable, observable and economically deployable. The field overlaps heavily with cloud architecture, DevOps, Site Reliability Engineering, MLOps and platform engineering.

You don't need to master everything before applying for a job. But understanding Linux, networking, Python, Docker, Kubernetes and at least one major cloud platform gives you a strong foundation. Then you need projects that prove you can actually build.
Here are five.

Project 1 — Generate Dockerfiles Using a Local LLM

Difficulty: Beginner

Tools: Ollama, Python, Docker and an open-weight model

This is a good first project because it connects AI with one of the foundations of modern infrastructure: containers. A Dockerfile tells Docker how to package an application and its dependencies into a reproducible container.

Your project is simple:

Give a local AI model a software project and ask it to inspect the files and generate an appropriate Dockerfile.

Start by installing Ollama, which lets you run supported open models locally.

Your Python program could collect useful project files such as package.json, requirements.txt or directory information and send that context to the local model.

Conceptually:

import requests

project_context = """
Project: Flask API
Python: 3.12
Dependencies:
flask
gunicorn
pandas

Application entry point: app.py
"""

prompt = f"""
Create a production-ready Dockerfile for this project.

Requirements:
- Use a small base image
- Do not run the application as root
- Cache dependencies efficiently
- Expose the correct port
- Use Gunicorn in production

Project:
{project_context}

Return only the Dockerfile.
"""

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "YOUR_LOCAL_MODEL",
        "prompt": prompt,
        "stream": False
    }
)

print(response.json()["response"])

Don't stop after generating the file.

Build it:

docker build -t ai-generated-app .

Run it.

Break it deliberately.

Give the model a bad Dockerfile and ask it to diagnose the failure. Then compare its output against Docker's official recommendations. Docker's Dockerfile best practices emphasise ideas including small trusted base images, rebuilding regularly and using multi-stage builds where appropriate.
What you learn: containers, local LLM inference, prompt design, dependency management and the basic relationship between AI applications and infrastructure.

Project 2 — Build a Production LLM Serving Stack on Kubernetes

Difficulty: Intermediate

Tools: vLLM, Kubernetes and Helm

Now stop treating an LLM as something running conveniently on your laptop.

Deploy one like a real service. vLLM is an open-source inference and serving engine designed for large language models. Its ecosystem includes an official production-stack project for deploying LLM serving infrastructure on Kubernetes.

Start with a local Kubernetes environment such as Minikube or kind if you don't have cloud resources.

1. Deploy a small model

2. Then expose an API endpoint

3. Once one replica works, create multiple replicas

4. Now generate concurrent requests

You want to observe: User requests → router → model replicas → GPU/compute → response

Next, deliberately kill one pod.

Does Kubernetes recreate it?

Increase traffic.

How are requests distributed?

Measure latency.

Watch memory consumption.

Then study KV cache behaviour. During LLM inference, key-value caches store intermediate attention data so the model doesn't repeatedly recompute previous tokens. Managing that cache becomes increasingly important as requests and context lengths grow.
The official vLLM Production Stack includes Kubernetes-native deployment, model-aware routing, observability and KV-cache-aware mechanisms.

What you learn: Kubernetes deployments, services, replicas, inference serving, request routing, model loading, Helm and the architecture behind production AI APIs.
This is where your portfolio begins looking like AI infrastructure rather than another chatbot project.

Project 3 — Build an End-to-End DevOps + AIOps System

Difficulty: Intermediate

Tools: Kubernetes, Prometheus, Grafana, logs and automation

Production infrastructure doesn't merely need to work. You need to know when it stops working. Build a small application consisting of several services. Deploy it to Kubernetes and install monitoring.

Track metrics such as:

CPU utilisation
Memory consumption
Request latency
Error rate
Requests per second
Pod restarts
GPU utilisation, where available.

Prometheus can collect metrics while Grafana visualises them.

Now deliberately cause problems.

Overload an API

Kill a pod

Create a memory-heavy process

Generate HTTP 500 errors

Your monitoring system should detect them

Then introduce the AI component

Instead of merely generating:

ALERT: API latency exceeded 2 seconds

Give an LLM relevant logs, metrics and deployment context and ask:

Summarise the incident, identify the most likely cause, show the evidence supporting your conclusion and recommend investigation steps. Do not execute changes.

Once that works reliably, experiment with bounded automation.

For example:

Monitoring detects failure → logs are collected → AI analyses incident → recommendation produced → human approves remediation.

That final human checkpoint is important. An AI model incorrectly diagnosing an incident is annoying. An autonomous AI with unrestricted production credentials acting on the wrong diagnosis can become a security incident itself.

What you learn: observability, incident response, metrics, logs, Kubernetes operations and the practical difference between DevOps and AIOps.

Project 4 — Build a GPU Autoscaling Inference System

Difficulty: Advanced

Tools: KServe, Kubernetes and GPUs

Now things become much more interesting.

Imagine you run an AI company.

At 2 a.m., 20 people use your model.

At 10 a.m., 5,000 people arrive.

Running enough GPUs for peak traffic all day wastes money. Running too few creates queues and slow responses.

You need autoscaling.

KServe provides Kubernetes-native model-serving infrastructure and supports autoscaling approaches for inference workloads.

Deploy a GPU-backed model-serving workload. Then create artificial traffic.

Start with:

10 concurrent users
↓
50 users
↓
100 users
↓
500 users

Monitor request latency, queue depth, GPU utilisation and replica count.

The goal is to observe your infrastructure responding automatically:

Traffic increases
      ↓
Inference demand rises
      ↓
Autoscaling condition triggers
      ↓
Additional replica starts
      ↓
Traffic distributed
      ↓
Demand falls
      ↓
Replica count decreases

Experiment with different scaling signals.

CPU utilisation alone may be a poor indicator for GPU inference. Queue depth, concurrent requests and inference-specific metrics can sometimes provide better signals.
KServe documents autoscaling for generative inference, including Kubernetes Horizontal Pod Autoscaler and KEDA-based approaches. KServe autoscaling documentation

What you learn: GPU workloads, model serving, concurrency, load testing, inference economics and autoscaling.

If you cannot access a GPU, don't abandon the project. Deploy a smaller CPU workload and reproduce the architecture. Simulate traffic and learn how Kubernetes responds. Then document which components you would change for GPU deployment.

An engineer who can explain the architecture clearly is already learning something valuable.

Project 5 — Build Scale-to-Zero GPU Inference With KEDA

Difficulty: Advanced

Tools: KEDA, Redis, Kubernetes, vLLM, GPUs and optionally GKE

Now solve an expensive AI infrastructure problem:

What happens when nobody is using your model?

Suppose your GPU costs money every hour.

At midnight there are zero requests. Why keep it running?

Scale-to-zero attempts to reduce the serving workload to zero when there is no demand and bring it back when requests arrive.

KEDA is a Kubernetes-based event-driven autoscaler that can scale workloads based on external signals.

A useful architecture is:

User request
     ↓
Redis queue
     ↓
KEDA watches queue
     ↓
Demand detected
     ↓
Inference workload scales up
     ↓
GPU/model becomes available
     ↓
Request processed
     ↓
Queue empties
     ↓
Idle period
     ↓
Workload scales toward zero

A reference implementation is available in GPU Autoscale Inference, which combines KEDA, Redis, Kubernetes, vLLM and GPU infrastructure.

Measure one thing carefully:

cold-start time

Scaling to zero saves money, but the next request may have to wait while a node starts, a container launches and a large model loads into GPU memory.

You are now dealing with a real infrastructure trade-off:

Cost vs latency.

Perhaps zero replicas are appropriate overnight.

Perhaps keeping one warm replica gives users a better experience.

Maybe popular models stay loaded while rarely used models scale to zero.

There isn't one correct answer.

That's what makes it architecture.

What you learn: event-driven architecture, queues, GPU economics, Kubernetes autoscaling, cold starts, capacity planning and production trade-offs.

What Should Your Portfolio Actually Show?

Don't simply put five GitHub repositories on your CV. Document your decisions.

For each project, explain:

Problem → architecture → implementation → failure encountered → measurement → improvement → cost implications.

1. Include an architecture diagram.

2. Include screenshots from Grafana.

3. Show load-test results.

4. Show what happened when you killed a pod.

5. Measure requests per second before and after optimisation.

6. Explain why you chose one autoscaling signal over another.

An employer hiring an infrastructure engineer is often more interested in how you think when systems fail than whether you followed a tutorial perfectly.

What Jobs Can These Skills Lead To?

You don't necessarily have to search exclusively for “AI Infrastructure Architect.”

Related roles include AI Infrastructure Engineer, ML Infrastructure Engineer, MLOps Engineer, Cloud Architect, Cloud Infrastructure Engineer, Platform Engineer, Site Reliability Engineer, GPU Infrastructure Engineer, Solutions Architect and AI Platform Engineer.

Compensation can be substantial in the United States, although salaries vary dramatically by employer, location, seniority and equity.

For example, current US job postings show infrastructure-oriented AI roles reaching well into six figures. OpenAI lists some infrastructure engineering positions with base compensation around $185,000–$385,000 plus equity, while Anthropic lists some infrastructure engineering roles around $320,000–$485,000 annual salary, depending on the position. These are examples from highly funded frontier-AI companies, not typical salaries across the entire profession.

The US Bureau of Labor Statistics provides a more conservative benchmark for adjacent architecture work: computer network architects had median annual pay of $130,390 in May 2024, with the highest 10% earning more than $198,030.

So rather than promising readers a $400,000 job, a sensible expectation is that cloud and AI infrastructure skills can lead into a broad family of well-paid engineering positions, with elite US AI companies paying substantially more for experienced specialists.

Why This Could Matter Even More for African Engineers

Africa currently has an infrastructure gap. But an infrastructure gap also creates an engineering problem that somebody must solve.

• Data centres need designing

• GPU clusters need operating

• AI models need serving

• Networks need configuring

• Power needs managing

• Cloud platforms need securing

• Inference needs optimising

• Companies need engineers who understand how all those layers interact

As more AI infrastructure comes to Lagos, Nairobi, Johannesburg, Cairo and other African technology centres, simply knowing how to call an AI API will not be enough.

The valuable engineers will increasingly be those who understand what happens underneath the API.

Africa's infrastructure constraints could even make these skills more valuable locally. An engineer deploying AI in a region with expensive compute, unreliable power or limited GPU availability has to care intensely about efficiency.

• Can a smaller model solve the problem?

• Can requests be batched?

• Can idle GPUs scale down?

• Can inference run locally?

• Can workloads move between regions?

• Can renewable energy reduce operating costs?

These questions connect directly with Afritech Connect's previous analysis of why Africa needs its own data centres but faces a major power problem.

You Don't Need a GPU Cluster to Start

This is perhaps the most important point for African students and developers.

1. Do not wait until you own an NVIDIA H100.

2. Start with your laptop.

3. Learn Linux.

4. Learn networking fundamentals.

5. Learn Python.

6. Containerise applications with Docker.

7. Run Kubernetes locally using kind or Minikube.

8. Deploy a small open model.

9. Create multiple replicas.

10. Break them.

11. Monitor them.

12. Build queues.

13. Generate artificial traffic.

14. Measure latency.

15. Study GPU architecture even when you don't yet have a GPU.

Cloud providers can be used later for short experiments when you genuinely need hardware—carefully monitoring costs so you don't accidentally leave expensive GPU instances running.

And read production repositories. Sometimes spending three hours understanding why an infrastructure project is architected a particular way teaches more than blindly copying its commands.

The five projects above form a natural progression:

Local AI → Containers → Kubernetes → Production serving → Observability → GPU autoscaling → Scale-to-zero.

By the final project, you are no longer merely asking an AI model questions. You are learning how to build the infrastructure that allows millions of other people to ask it questions reliably.

That distinction could become extremely valuable. The AI boom is creating enormous attention around models, agents and applications. But every successful AI product eventually runs into the same physical reality: models require compute, compute requires infrastructure and infrastructure requires people who know how to operate it.

As billions of dollars begin flowing into AI infrastructure globally—and more of that investment reaches Africa—the continent will need more than GPUs.

It will need engineers who know what to do with them.

Alternatively, if you would like to learn alongside industry experts from silicon valley. I would advise you should look into udacity course works on architech. However, udacity are priced to be expensive but it worths if you can afford it because all their course works have real world hands-on practical projects to get you used to the field. They also offer scholarships occasionally. You can also look into youtube videos to make your learning experiences seamlessly.

Read more: Best Laptops for Learning Cloud and AI Infrastructure Architecture: 3 Picks From ₦500K to ₦2M+
Read more: How to Use AI to Improve Productivity at Work Without Creating More Work
Read more: Andrew Ng’s AI Skills Map: 6 Skills African Should Learn to Build Real AI Products Part 2
|Read more: ChatGPT Removed Text Chat Limits for Free Users: What Nigerians and Africans Need to Know