Processes, Threads & Concurrency

Embedded and systems code frequently has more than one thing happening at once -- an interrupt firing mid-function, a second thread, a signal handler -- and unlike a single-threaded script, ordering is no longer something you can assume.

A process has its own memory space; a thread shares memory with every other thread in the same process. That sharing is the whole point of threads -- fast communication with no copying -- and it's also the whole source of the danger: any thread can read or write any other thread's data structures with no OS-enforced boundary between them the way there is between processes.

A race condition happens when the correctness of a result depends on the timing of two or more threads, and that timing isn't guaranteed. The textbook example: two threads both do "read counter, add one, write counter" on a shared variable with no protection. If both read the same starting value before either writes back, one increment is silently lost -- and this happens rarely enough in testing that it's a classic "works on my machine, fails in the field" bug.

A mutex (mutual exclusion lock) makes a region of code atomic with respect to other threads: only one thread can hold it at a time, so code between lock and unlock runs as if nothing else could interleave with it. A semaphore is more general -- it holds a count rather than a single lock/unlock state, so it can allow up to N threads into a region at once, or be used to signal between threads rather than just protect data. Get the discipline wrong -- forget to unlock, unlock twice, lock in an order that another thread locks in reverse -- and you get a deadlock: two threads each holding a lock the other needs, both waiting forever.

Priority inversion is the specific, nasty case where a low-priority thread holds a lock a high-priority thread needs, and a medium-priority thread that needs neither lock preempts the low-priority one — so the high-priority thread waits not just for the lock, but indirectly for a thread that outranks it in priority to even get scheduled. This is famous partly because it took down the Mars Pathfinder rover in 1997, and the standard fix — priority inheritance, where the low-priority thread temporarily inherits the waiting thread's priority while it holds the lock — is a concrete, nameable answer interviewers listen for.

Inter-process communication (IPC) is how separate processes, which don't share memory, exchange data anyway: pipes, message queues, shared memory segments (which reintroduce the same race conditions threads have, deliberately, in exchange for speed), and sockets — the same mechanism used for network communication, which also works perfectly well between two processes on the same machine.