Practice splitting a problem into many independent pieces of work and spotting when tasks depend on each other. You will use everyday examples and small code exercises to separate parallel work from serial bottlenecks.

Trace GPUs from graphics pipelines to general-purpose computing, CUDA, OpenCL, mobile GPUs, AI accelerators, ray-tracing hardware, and today’s cloud GPU clusters. The chapter connects old design choices to the tools and limits current GPU programmers face.

Meet threads, blocks, grids, warps, streaming multiprocessors, registers, shared memory, global memory, and device-host transfers. You will map a simple vector operation onto GPU hardware without hiding behind magic terminology.

Install a working toolchain, compile a small GPU program, run it, and check that the device is visible. The focus is on CUDA first because it is the most common learning path, while keeping the ideas portable.

Write kernels that run thousands of threads, pass data to the device, launch work, and copy results back. You will build vector addition and simple array transforms while learning launch configuration and error checking.

Handle device allocation, host allocation, memory copies, pinned memory, and common ownership mistakes. You will practice drawing where data lives at each moment so bugs become easier to find.

Use thread and block indexes to cover one-dimensional, two-dimensional, and three-dimensional data. You will write kernels for images and matrices while avoiding off-by-one and out-of-bounds errors.

Measure GPU code with timers, profilers, occupancy reports, and memory-throughput counters instead of guessing. You will compare CPU and GPU versions honestly, including transfer costs and warm-up effects.

Recognize coalesced access, strided access, alignment, cache behavior, and memory bandwidth limits. You will rewrite kernels so neighboring threads fetch neighboring data and wasted memory traffic drops.

Use shared memory to reuse data inside a block and reduce slow global memory reads. You will build tiled matrix operations and learn how bank conflicts and synchronization affect speed.

Use thread barriers, atomics, reductions, scans, and safe update patterns when many threads work on shared results. You will build sums, histograms, and prefix operations without race conditions.

Read PTX/SASS at a practical level, manage registers, avoid excessive spilling, and reason about occupancy without chasing a single magic number. You will connect source code choices to what the GPU actually schedules.

Use floating-point types, integer types, rounding behavior, fused operations, and error checks wisely. You will see why GPU results can differ from CPU results and how to decide whether the difference matters.

Build common GPU patterns: map, stencil, reduction, scan, gather, scatter, compact, sort, and histogram. These patterns become reusable building blocks for simulations, image processing, analytics, and machine learning.

Use libraries such as cuBLAS, cuFFT, cuSPARSE, Thrust, CUB, and vendor-neutral alternatives before writing custom kernels. You will decide when a library call is safer, faster, and easier to maintain than hand-written code.

Handle asynchronous copies, streams, events, overlap between compute and transfer, and pipeline-style execution. You will make independent work run concurrently while keeping correctness visible.

Use unified memory, prefetching, memory advice, and oversubscription carefully. You will see when managed memory makes code simpler and when page migration can quietly destroy performance.

Find illegal accesses, race conditions, uninitialized data, launch failures, numerical surprises, and performance regressions. You will use sanitizers, debuggers, assertions, deterministic tests, and small reproducible cases.

Work with textures, surfaces, interpolation, image layouts, and GPU-friendly formats. You will connect compute kernels with image processing tasks where spatial locality and sampling matter.

Use warp-level primitives, shuffles, ballots, cooperative groups, and warp-aware reductions. You will write faster kernels by coordinating threads at the level the hardware actually executes.

Use mixed precision, tensor cores, matrix fragments, and accumulation rules for high-throughput linear algebra. You will learn where tensor cores shine, where accuracy can suffer, and how modern AI workloads use them.

Write and tune GPU code for convolutions, attention, normalization, embedding lookups, and batch processing. This chapter connects core GPU skills to the workloads behind deep learning and large language models.

Use Triton-style kernel programming and compiler-driven GPU tools to express tiled parallel algorithms with less low-level boilerplate. You will compare generated kernels with CUDA kernels and see where these tools are strong or limited.

Capture repeated GPU work with CUDA Graphs and reduce launch overhead in training, inference, simulation, and real-time systems. You will turn a sequence of kernels and copies into a reusable execution graph.

Program more than one GPU with peer-to-peer copies, NVLink, topology awareness, partitioning, and workload distribution. You will build a simple multi-GPU pipeline and measure whether scaling is actually happening.

Use collective communication libraries such as NCCL for all-reduce, broadcast, gather, and distributed training patterns. You will connect GPU kernels with the communication steps needed by clusters.

Run GPU workloads in containers, schedule them on shared machines, and use cloud instances responsibly. The chapter covers drivers, runtimes, device plugins, quotas, cost control, and reproducible environments.

Write GPU code across CUDA, HIP, OpenCL, SYCL, Vulkan compute, Metal, and portability layers without pretending they are identical. You will learn how memory models, tooling, platform support, and performance tuning differ.

Use WebGPU and browser-safe compute to run parallel workloads outside native desktop or server apps. You will build a small compute example and see how safety, portability, and shader languages shape the design.

Use ray-tracing cores, acceleration structures, shader binding tables, and compute interop for rendering and simulation tasks. You will see how hardware ray tracing changes real-time graphics and non-graphics workloads.

Design kernels for graphs, sparse matrices, irregular meshes, particles, and branching-heavy workloads. You will practice choosing layouts and algorithms that reduce divergence and wasted memory traffic.

Connect GPU kernels to CPU services, storage, networking, message queues, and user-facing applications. You will design data flow so the GPU speeds up the whole system, not just one isolated function.

Follow a complete job from a performance problem to a shipped GPU solution: define the target, choose libraries or custom kernels, profile, optimize, test, package, deploy, and monitor. This chapter ties together technical choices and practitioner judgment.

Protect shared GPU systems from driver drift, unsafe dependencies, data leaks, side channels, and untrusted code. You will also cover reproducible builds, model and kernel artifacts, and the risks that appear when GPUs become production infrastructure.

Compare roles in GPU software, HPC, graphics, embedded systems, AI infrastructure, scientific computing, and performance engineering. You will plan portfolio projects, proof-of-skill benchmarks, useful certifications or degrees, and habits for staying current as hardware changes.