IPC in OS: Pipes, Shared Memory, Message Queues

Processes are isolated on purpose: each has its own address space, so one cannot read or overwrite another's memory. But programs often need to cooperate. A shell connects ls to wc, a web server hands requests to worker processes, a browser's tabs talk to its main process. Inter-process communication (IPC) is the set of kernel-provided channels that let processes exchange data and signal each other without giving up that isolation. This note covers the two underlying models and the mechanisms built on them.

Why processes communicate

Cooperating processes are used for information sharing (several programs reading one data set), computation speed-up (splitting work across processes on several cores), modularity (building a system from separate programs, as Unix pipelines and microservices do) and convenience (a user editing, compiling and printing at once). Threads of one process share memory already, so they need synchronization but not IPC; separate processes need both.

Shared memory vs message passing

Every IPC mechanism follows one of two models.

In shared memory, the processes ask the kernel to map one region of physical memory into each of their address spaces. After that setup, they exchange data by ordinary reads and writes, with no kernel involvement, which makes it the fastest form of IPC. The processes themselves are responsible for not writing at the same time: a race on shared memory is exactly the problem of Process Synchronization, usually solved with a semaphore or mutex placed in the shared region.

In message passing, the processes call send(message) and receive(message), and the kernel copies the message from the sender to the receiver. It costs a system call and a copy per message, but there is nothing to corrupt, it is easier to get right, and the same model works between machines.

1. Shared memory: the kernel maps one physical page into both address spaces once, and afterwards A stores and B loads with ordinary instructions. 3 messages cost no system calls and no copies, and nothing stops A and B colliding.
2. Message passing: every send is a system call that copies the message into a kernel queue, and every receive a second one that copies it out. 3 messages cost 6 system calls and 6 copies, but the kernel orders and synchronizes them.

Design choices in message passing

• Direct or indirect. In direct communication, a process names its partner: send(P, msg). In indirect communication, messages go to a mailbox (also called a port), and any process sharing the mailbox can receive them.
• Blocking or non-blocking. A blocking (synchronous) send waits until the message is received; a non-blocking send returns at once. A blocking receive waits until a message arrives; a non-blocking receive returns a message or nothing. When both sides block, the exchange is a rendezvous.
• Buffering. The link's queue has zero capacity (the sender must wait for the receiver, a rendezvous), bounded capacity (the sender waits only when the queue is full) or, in theory, unbounded capacity (the sender never waits).

Pipes

An ordinary pipe (anonymous pipe) is a one-way byte stream inside the kernel. pipe(fd) creates it and returns two file descriptors: fd[0] is the read end and fd[1] the write end. Since it has no name, other processes can reach it only by inheriting the descriptors, so pipes connect related processes, typically a parent and the child it forks.

#include <stdio.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void) {
    int fd[2];                          /* fd[0]: read end, fd[1]: write end */
    if (pipe(fd) == -1) return 1;
    if (fork() == 0) {                  /* child: the reader */
        char buf[32];
        close(fd[1]);                   /* close the end it does not use */
        ssize_t n = read(fd[0], buf, sizeof buf - 1);
        buf[n > 0 ? n : 0] = '\0';
        printf("child read: %s\n", buf);
        return 0;
    }
    close(fd[0]);                       /* parent: the writer */
    write(fd[1], "hello", 5);
    close(fd[1]);                       /* the reader now sees end of file */
    wait(NULL);
    return 0;
}

1. pipe(fd) made a kernel buffer with a read end and a write end, and fork() gave the child both. Each process closed the end it does not use: the parent writes, the child reads. The buffer holds 8 bytes here, 64 KB on Linux.
2. The parent writes 11 bytes. 8 fit and the buffer is full, so the write blocks with 3 bytes still to go: a pipe is a bounded buffer and the kernel makes the writer wait.
3. The child reads 5 bytes and gets "hello", the oldest bytes first. That frees 5 bytes of space.
4. The writer wakes, puts its last 3 bytes in and the call returns 11. The child never sees where one write ended and the next began; a pipe carries only bytes.
5. The parent closes the write end. No process holds it now, which the reader will notice only once the buffer is empty.
6. The child asks for 16 bytes and gets the 6 that are there: a read returns what is available rather than waiting to fill its buffer.
7. The buffer is empty and every write end is closed, so read returns 0, end of file. Had the child kept its own copy of fd[1] open, this read would block for ever.

Rules worth knowing:

• Each process should close the end it does not use. A reader sees end of file only when every write end is closed; a forgotten write end in the reader itself means it waits forever.
• Writing to a pipe whose read ends are all closed raises the signal SIGPIPE in the writer.
• The pipe has a fixed kernel buffer (64 KB by default on Linux). A writer blocks when it is full and a reader blocks when it is empty, which is the bounded buffer of Classic Synchronization Problems, done by the kernel.
• A shell pipeline such as ls | wc -l is two processes joined by an ordinary pipe: the shell creates the pipe, forks twice and connects one process's standard output to the other's standard input:

1. The shell calls pipe(): descriptors are handed out lowest free first, and 0, 1 and 2 are taken, so the read end is 3 and the write end 4.
2. It forks twice. Each child starts with a copy of the shell's table, so both hold 3 and 4 as well as the terminal on 0, 1 and 2.
3. Child 1 calls dup2(4, 1), child 2 dup2(3, 0); both close 3 and 4 and exec, and the shell closes its copies too. Now ls writes its standard output into the pipe and wc reads its standard input from it, and only ls holds a write end.

For two-way communication, use two pipes, one in each direction.

Named pipes (FIFOs)

A named pipe, or FIFO, is a pipe with a name in the file system, created with mkfifo. Any process with permission can open the name, so FIFOs connect unrelated processes on the same machine. The name persists after the processes exit, until it is deleted, although the data inside does not. On Unix a FIFO is still one-way and byte-oriented, and opening one end blocks until the other end is opened too. Windows named pipes are different: they can be two-way and can even work between machines on a network.

mkfifo /tmp/jobs          # create the FIFO
cat /tmp/jobs &           # a reader waits on it
echo "build" > /tmp/jobs  # a writer in another process; the reader prints build

Message queues

A message queue is a linked list of messages kept by the kernel. Unlike a pipe's byte stream, it preserves message boundaries: each receive returns one whole message. System V queues (msgget, msgsnd, msgrcv) give each message a type, so a receiver can ask for messages of one type; POSIX queues (mq_open, mq_send, mq_receive) give each a priority, and the highest-priority message is received first. A queue exists until it is explicitly removed, even if no process has it open, so a sender and a receiver need not run at the same time.

1. The same two writes go into a pipe and a message queue. The pipe keeps only bytes, so one read can return "abcd" with the seam gone; the queue keeps each message whole, and each receive returns exactly one.
2. A POSIX queue also orders by priority: "stop" was sent second but has priority 5, so it is received first, and the two priority-1 messages follow in the order they were sent.

Shared memory segments

With POSIX shared memory, one process creates a named object and both map it:

#include <fcntl.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void) {
    int fd = shm_open("/demo", O_CREAT | O_RDWR, 0600);   /* create or open by name */
    ftruncate(fd, 4096);                                   /* set its size */
    char *p = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    strcpy(p, "hello from shared memory");                 /* a plain memory write */
    munmap(p, 4096);
    close(fd);
    return 0;               /* a reader maps "/demo" the same way; shm_unlink removes it */
}

Once mapped, the data never passes through the kernel again. The segment outlives the processes until shm_unlink (or System V's shmctl with IPC_RMID) removes it.

Sockets

A socket is an endpoint for two-way communication, identified on a network by an IP address and a port (for example 192.168.1.10:8080). Sockets are the only common IPC mechanism that works between machines, and they are the foundation of client-server systems: web servers, databases and chat applications all use them.

• Stream sockets (TCP) give a reliable, ordered byte stream over a connection.
• Datagram sockets (UDP) send independent messages that may be lost or reordered.
• Unix domain sockets use a file-system path instead of an address and work only on one machine, faster than TCP over the loopback interface; many local services, such as database servers, offer one.

A server calls socket, bind, listen and accept; a client calls socket and connect; both then send and recv:

1. The server makes a socket, descriptor 3, binds it to port 8080, marks it listening and calls accept(), which waits for a client.
2. The client makes its own socket and calls connect(). The kernels exchange the TCP handshake, and accept() returns a new descriptor, 4, for this one connection; 3 goes on listening for others.
3. Both sides now send and recv on their connected descriptors, 3 and 4: a reliable two-way byte stream that works the same whether the two processes share a machine or not.

Signals

A signal is an asynchronous notification sent to a process to tell it that an event happened. It carries no data beyond its number. Signals come from the kernel (an illegal memory access raises SIGSEGV), from other processes (kill(pid, SIGTERM)) or from the terminal (Ctrl+C sends SIGINT).

A process can accept the default action, ignore the signal, or install a handler that runs when it arrives:

#include <signal.h>
#include <stdio.h>
#include <unistd.h>

static volatile sig_atomic_t stop = 0;
static void on_sigint(int sig) { (void)sig; stop = 1; }    /* keep handlers tiny */

int main(void) {
    struct sigaction sa = {0};
    sa.sa_handler = on_sigint;
    sigaction(SIGINT, &sa, NULL);   /* Ctrl+C now runs on_sigint instead of killing us */
    while (!stop) pause();          /* sleep until a signal arrives */
    printf("caught SIGINT, shutting down cleanly\n");
    return 0;
}

Standard signals are not queued: if the same signal arrives twice before it is handled, the process may see it once. Signals suit notifications such as "reload your configuration" or "a child exited", never bulk data.

Comparing the mechanisms

Common mistakes

• Saying shared memory needs no synchronization because it is fast. Speed comes precisely from the kernel staying out, so the processes must coordinate themselves.
• Saying an ordinary pipe can connect any two processes. Without a name, only processes that inherit its descriptors can use it.
• Forgetting to close unused pipe ends, so the reader never sees end of file.
• Treating a pipe as two-way. A Unix pipe is one-way; use two pipes or a socket pair.
• Claiming SIGKILL can be handled. SIGKILL and SIGSTOP can be neither caught nor ignored.
• Assuming message boundaries survive in a pipe or TCP stream. Two writes may arrive as one read; message queues and UDP keep boundaries.

Interview questions

Why is shared memory faster than message passing? After the region is mapped, processes exchange data with ordinary memory instructions, without system calls or copies. Message passing enters the kernel and copies the data twice, from the sender into a kernel buffer and from there to the receiver, for every message.

How does a shell implement ls | wc -l? It creates a pipe, forks a child for ls that makes the pipe's write end its standard output, and forks a child for wc -l that makes the read end its standard input. Each child closes the ends it does not use and calls exec on its program; the shell closes both ends and waits.

When would you use a socket instead of a pipe? When the processes may be on different machines, when they are unrelated and need a two-way channel, or when a server must accept connections from many clients. For two related processes on one machine, a pipe is simpler.

What happens when a process writes to a full pipe, or reads from an empty one? The writer blocks until a reader removes data, and the reader blocks until data arrives. If all write ends are closed, a read on an empty pipe returns end of file instead of blocking.

What is the difference between blocking and non-blocking message passing? A blocking send waits until the message is received and a blocking receive waits until a message arrives. Non-blocking calls return at once, with the message or with nothing, and the process checks again later.

Can signals be used to send data between processes? Not in any practical sense. A standard signal carries only its number and does not queue, so it is a notification mechanism. Data should go through a pipe, queue, socket or shared memory, with a signal at most announcing that it is there.

You have reached the end of the operating systems notes. Go back to the Operating Systems notes to revise any topic, then take the Operating Systems (Basic) and Operating Systems (Intermediate) skill tests.