Graphics Processing lessons

Many small cores

A CPU has a few powerful cores. Each one is good at long chains of steps, where every step depends on the one before.

A CPU with 4 cores and a GPU with 64 cores shade the same 192 pixel image, counting the steps each one takes.

A GPU has thousands of simpler cores. They all run the same small program at once, each on its own pixel.

A full HD screen has over two million pixels. Each pixel can be worked out on its own, so the GPU does thousands at the same time.

Practice: CPU or GPU?

Sort tasks into the ones that suit a GPU and the ones that suit a CPU.

Pixels and the frame buffer

Each pixel stores its colour as three numbers: red, green and blue. Each number runs from 0 to 255, so it fits in one byte.

A 6 by 4 pixel image beside its memory: each pixel in turn shows its three bytes and where they sit.

The frame buffer is the block of memory that holds every pixel, row after row. The GPU draws each new frame into it, and the screen reads it about 60 times a second.

A full HD screen is 1920 \times 1080 pixels. At 3 bytes each, one frame needs 6\ 220\ 800 bytes, about 6 MB.

Practice: Size a frame buffer

Type how many bytes a small screen's frame buffer needs, from its width, height and bytes per pixel.

Triangles, vertices and meshes

A vertex is a corner point with a position (x,\ y,\ z). Three vertices make a triangle, the one shape every GPU is built to draw.

Three meshes turn in 3D, drawn as their triangles, with live vertex and triangle counts.

A mesh is many triangles joined edge to edge. Neighbouring triangles share vertices, so a cube needs only 8 vertices for its 12 triangles.

A triangle is always flat, which makes it quick to fill. More triangles give smoother curves, but they cost more work.

Practice: Count the triangles

Count the triangles or vertices in small flat meshes: grids of squares and fans.

The rendering pipeline

The rendering pipeline is the fixed order of steps that turns triangles into pixels. Every frame goes through it.

The six pipeline stages light up in turn, each saying what it does to the data.

Two of the stages run small programs you write, called shaders. The other stages are built into the GPU.

Coordinate spaces

A model is built around its own centre, in model space. To place it in a scene, its vertices are moved into world space.

One vertex of a small ship is followed from model space to world space, view space and screen space, with its coordinates worked out at each step.

Next, everything is moved so the camera sits at the centre, in view space. Last, positions become pixels on the screen, in screen space, where y counts down from the top.

Each step is the same maths applied to every vertex. Doing it is the vertex shader's job.

Practice: Put it in order

Put the six pipeline stages in order, then the four spaces a vertex passes through.

Transforms with matrices

A transform changes every vertex in the same way. To translate a shape, you add the same amount to every vertex.

To scale or rotate it, you multiply each vertex by a matrix, a small grid of numbers. Each row of the matrix makes one new coordinate.

1. x' = x\cos\theta - y\sin\theta (row 1 gives the new x)
2. y' = x\sin\theta + y\cos\theta (row 2 gives the new y)

A triangle is translated, scaled and rotated on a grid, with the matrix sum for one vertex worked out each time.

In a vertex shader this is one line, such as gl_Position = uMatrix * vec4(aPosition, 1.0);. The GPU runs it once for every vertex.

Practice: Transform a vertex

Type where a vertex lands after a translate, a scale, or a rotation of 90, 180 or 270 degrees.

Rasterisation

Rasterisation finds the pixels a triangle covers. A pixel counts when its centre is inside the triangle.

A scan tests every pixel centre against the triangle's three edges and lights the ones inside, counting fragments.

To test a centre, the GPU checks which side of each edge it is on. If it is inside all three edges, the pixel becomes a fragment.

A fragment is a covered pixel that is still waiting for its colour. Real GPUs test many centres at once.

Blending across the triangle

Each vertex can carry extra values, such as a colour. The rasteriser blends them for every fragment inside the triangle.

A point moves around inside a red, green and blue triangle; the three corner weights and the blended colour update as it goes.

The blend uses three weights, one per corner, that add up to 1. A fragment close to corner A gets a big weight for A.

Blending: \text{colour} = w_A A + w_B B + w_C C

Practice: Which pixels light up?

Tap the marked pixels whose centres fall inside a triangle on a pixel grid.

The fragment shader

The fragment shader is a small program that runs once for every fragment. It gives the final colour as four numbers: red, green, blue and alpha (opacity).

Four short GLSL fragment shaders run on every pixel of a 16 by 12 grid; one pixel's inputs and output are shown.

Shaders for WebGL are written in GLSL. In GLSL, each colour channel runs from 0.0 to 1.0, not 0 to 255.

vec4 holds four numbers. The shader puts its answer in gl_FragColor.

Practice: Write a fragment colour

Type the vec4 that makes a named colour, with a live preview of the colour typed.

Depth testing

Triangles often overlap on screen. The nearest one must win, whatever order they are drawn in.

A near and a far triangle are drawn into a colour buffer and a depth buffer, with the depth test on and then off.

The depth buffer stores one depth for every pixel, where a smaller number is nearer. A new fragment only replaces a pixel if it is nearer than the depth stored there.

This check is the depth test. Without it, whatever is drawn last ends up on top.

Practice: Which fragment wins?

Given what a pixel holds and a new fragment's depth, pick what the pixel shows after the depth test.

Textures and UV coordinates

A texture is an image wrapped onto a surface. Each pixel of a texture is called a texel.

Each vertex gets UV coordinates that pin it to a spot on the texture. u runs across from 0 to 1, and v runs up from 0 to 1.

A triangle on screen is filled from a small texture; one fragment's blended UV and the texel it reads are traced across.

The rasteriser blends the UVs for each fragment. The fragment shader then reads the texel at that spot, which is called sampling.

Practice: Sample the texture

Tap the texel a 4 by 4 texture returns for a given UV coordinate.

Diffuse lighting

A surface is brightest when it faces the light. Its normal \class{ix-c}{N} points straight out of it, \class{ix-a}{L} points at the light, and both have length 1.

Diffuse light: \text{brightness} = \max(0,\ \class{ix-c}{N}\cdot \class{ix-a}{L})

The dot product \class{ix-c}{N}\cdot \class{ix-a}{L} = \class{ix-c}{N_x}\class{ix-a}{L_x} + \class{ix-c}{N_y}\class{ix-a}{L_y} measures how much the surface faces the light. Facing it gives 1, side on gives 0, and facing away is held at 0.

A light circles a sphere lit pixel by pixel; beside it, a flat surface shows N, L and their dot product at the same angle.

Practice: Light a surface

Work out a surface's diffuse brightness from its normal and the light direction.

Graphics APIs

Programs do not talk to the GPU directly. They use a graphics API, a set of commands for sending triangles, textures and shaders.

An app's commands pass through each of five APIs to the driver and the GPU, with where each API runs and its shading language.

WebGL and WebGPU run in web browsers. Vulkan runs on many systems, DirectX on Windows and Xbox, and Metal on Apple devices.

Each API has its own shading language, but the pipeline underneath is the same. What you learned here carries across all of them.

Practice: Match the API

Match each graphics API to where it runs, then to its shading language.