Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

The Linux Interview Cookbook: A Practical Guide to Master Your Linux Skills

Book Cover

whoami

Salem! (kazakh version of “Hi”) I’m Maxat Akbanov — a DevOps Engineer based in the beautiful city of Almaty, Kazakhstan 🇰🇿. I’ve been working in tech for a while now, and somewhere along the way I became deeply passionate about cloud infrastructure, automation, and making systems that just… work reliably at scale.

My journey into IT field actually started back in my school days at the Zhautikov Republican Physics and Mathematics School (Fizmat), a place famous for shaping tech minds. While most kids were just playing video games, I found myself completely fascinated by how computer networks functioned and how software was built. I spent my spare time devouring books on Cisco networking and teaching myself the fundamentals of programming using Pascal and C. That early curiosity at Fizmat completely set the trajectory for my entire career.

Eventually, that passion for systems evolved into a deep focus on security. I studied Information and Systems Security for my Bachelor’s at the National Technical University of Ukraine in Kyiv, then went on to do my Master’s in Cyber Security at the University of York. During that time I got to dig deep into real-world threats — I even published a few research papers on the WannaCry ransomware. It was intense, but it gave me a perspective on infrastructure that I carry to this day: security isn’t an afterthought, it’s the foundation.

After university, I worked across all kinds of environments — from large Kazakhstani enterprises like Kazakhtelecom, to early-stage startups like ioka.kz, to globally recognized software companies like EPAM and DataArt. Each one taught me something different about how teams build and ship software. I also spent time as an independent consultant, helping clients design and support their cloud workloads on AWS — things like containerized services on EKS and ECS. Working alone, in small teams, in big distributed ones — it’s all shaped how I think about DevOps.

These days I channel everything I’ve learned about Linux into this book. I write articles on LinkedIn too, and I try to stay connected with the local tech community — attending conferences and giving talks at the AWS Cloud Club here in Almaty.

Outside of work, you’ll find me playing football or basketball, swimming, hiking in the mountains around Almaty, playing chess, or getting way too invested in an online strategy games. Life’s good.

I speak “Tech,” but I also speak human. My native tongue is Kazakh, and I’m fluent in Russian and English, with Spanish and Turkish currently “loading” in my spare time. Feel free to message me in any language — I promise to try and understand you. If my reply comes back as a mix of five different dialects and a bit of Bash script, just know I’m trying my best!

See you! / Көріскенше! / До встречи! / ¡Hasta luego! / Görüşürüz!
Maxat 😉

P.S. Technology moves fast. If you find an outdated command, concept, diagram or a better way to structure a manifest, section, or chapter, please reach out via my GitHub, Linkedin or email. Your feedback helps make this book better!

Support & Sponsorship

As an independent author, I rely on the support of the community to keep these resources updated and accessible.

If you find this book valuable, I welcome opportunities for support and collaboration to help expand and improve future editions.

I’m open for any of these opportunities:

  • Sponsorship and funding
  • Technical writing collaborations
  • Content review and expert contributions
  • Educational or industry partnerships

If you are interested in working together, please reach out with a brief proposal or introduction at:

maxat.akbanov@protonmail.com

Support for this project helps cover research and hosting costs. You can provide support through .

ko-fi

Thank you for supporting accessible technical knowledge.

Preface

Chapter 1: Permissions

Interview Questions

This chapter answers to the following questions:

Linux File Permissions

Every file and directory in Linux has an associated set of permissions that control who can read, write, or execute it. Understanding this system is fundamental to Linux administration and almost always comes up in technical interviews.


The Permission Model

Linux uses a discretionary access control (DAC) model. Each file has three permission classes:

ClassApplies to
User (u)The file’s owner
Group (g)Members of the file’s group
Other (o)Everyone else

Each class has three permission bits:

PermissionSymbolOctalOn a fileOn a directory
Readr4View file contentsList directory contents (ls)
Writew2Modify file contentsCreate, delete, rename files inside
Executex1Run as a programEnter the directory (cd)

rwx_permissions


Reading the Permission String

Run ls -l to see permissions:

-rwxr-xr--  1  alice  devs  4096  May 30 10:00  script.sh

Break down the first field, -rwxr-xr--:

- rwx r-x r--
│ │   │   └── Other:  read only
│ │   └────── Group:  read + execute
│ └────────── User:   read + write + execute
└──────────── File type: - (regular file)

File type characters:

SymbolType
-Regular file
dDirectory
lSymbolic link
cCharacter device
bBlock device
pNamed pipe (FIFO)
sSocket

Octal (Numeric) Notation

Each permission class is represented as a 3-bit binary number, summed into a single octal digit:

rwx = 4+2+1 = 7
r-x = 4+0+1 = 5
r-- = 4+0+0 = 4

So -rwxr-xr-- = 754.

Common permission values:

OctalBinarySymbolicMeaning
777111 111 111rwxrwxrwxFull access for everyone
755111 101 101rwxr-xr-xOwner full; others read/exec
644110 100 100rw-r--r--Owner read/write; others read
600110 000 000rw-------Owner read/write only
700111 000 000rwx------Owner full, no one else

Changing Permissions: chmod

Symbolic mode — readable and expressive:

chmod u+x script.sh        # add execute for owner
chmod g-w file.txt         # remove write from group
chmod o=r file.txt         # set other to read-only exactly
chmod a+r file.txt         # add read for all (a = ugo)
chmod u+x,g-w file.txt     # multiple changes at once

Numeric mode — precise and scriptable:

chmod 755 script.sh        # rwxr-xr-x
chmod 644 config.conf      # rw-r--r--
chmod 600 ~/.ssh/id_rsa    # rw------- (required by SSH)

Recursive:

chmod -R 755 /var/www/html

💡 Interview tip: Prefer numeric mode in scripts for clarity and predictability; prefer symbolic mode when doing targeted, additive changes interactively.


Changing Ownership: chown and chgrp

chown alice file.txt             # change owner to alice
chown alice:devs file.txt        # change owner and group
chown :devs file.txt             # change group only
chgrp devs file.txt              # change group (equivalent)
chown -R alice:devs /srv/app     # recursive

Only the root user (or a process with CAP_CHOWN capability) can change a file’s owner. A regular user can only change the group of a file they own, and only to a group they belong to.


Special Permission Bits

Beyond rwx, there are three special bits that modify execution behavior:

setuid (SUID) — octal 4000

When set on an executable file, the process runs with the file owner’s privileges instead of the invoking user’s.

chmod u+s /usr/bin/passwd   # symbolic
chmod 4755 /usr/bin/passwd  # numeric

In ls -l output, the owner execute bit shows s (or S if execute is not set):

-rwsr-xr-x  root  root  /usr/bin/passwd

passwd is the classic example: a normal user needs to write to /etc/shadow (owned by root), so the binary runs as root via SUID.

On a directory, SUID has no standard effect on Linux.

setgid (SGID) — octal 2000

On an executable file, the process runs with the file’s group privileges.

On a directory, new files created inside inherit the directory’s group instead of the creator’s primary group — essential for shared project directories.

chmod g+s /srv/shared        # symbolic
chmod 2775 /srv/shared       # numeric

In ls -l, the group execute bit shows s:

drwxrwsr-x  alice  devs  /srv/shared

Sticky Bit — octal 1000

On a directory, only the file’s owner, the directory’s owner, or root can delete or rename files within it — even if others have write permission on the directory.

chmod +t /tmp               # symbolic
chmod 1777 /tmp             # numeric

In ls -l, the other execute bit shows t:

drwxrwxrwt  root  root  /tmp

/tmp is the canonical example: world-writable but protected so users can’t delete each other’s files.

Summary of special bits:

BitOctalOn fileOn directory
SUID4000Run as file ownerNo standard effect
SGID2000Run as file groupNew files inherit directory group
Sticky1000No standard effectOnly owner can delete/rename files

suid_sgid


The umask

The umask defines which permission bits are removed from the default when a new file or directory is created.

Default creation modes:

  • Files: 0666 (no execute by default)
  • Directories: 0777

With a umask of 0022:

  • Files created as: 0666 - 0022 = 0644 (rw-r--r--)
  • Directories created as: 0777 - 0022 = 0755 (rwxr-xr-x)
umask           # display current umask
umask 0027      # set new umask (files: 0640, dirs: 0750)
umask -S        # display in symbolic form (u=rwx,g=rx,o=)

The umask is subtracted bitwise, not arithmetically. Think of it as a mask of bits to clear, not a number to subtract.


Access Control Lists (ACLs)

Standard Unix permissions allow only one owner and one group per file. ACLs extend this by allowing per-user and per-group rules.

# View ACLs
getfacl file.txt

# Grant bob read+write
setfacl -m u:bob:rw file.txt

# Grant the ops group read-only
setfacl -m g:ops:r file.txt

# Remove bob's ACL entry
setfacl -x u:bob file.txt

# Remove all ACL entries
setfacl -b file.txt

When ACLs are present, ls -l shows a + after the permission string:

-rw-rw-r--+  alice  devs  file.txt

Scenario: SSH Key Rejected — “Unprotected Private Key”

The Situation

You have generated an SSH key pair and are trying to connect to a remote server:

ssh -i ~/.ssh/id_rsa alice@remote-server.example.com

Instead of connecting, you receive this error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/alice/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/alice/.ssh/id_rsa": bad permissions
alice@remote-server.example.com: Permission denied (publickey).

ssh_key_scenario


Diagnosis

SSH refuses to use any private key file that other users can read. The error message tells you exactly what is wrong: the key file at ~/.ssh/id_rsa has permissions 0644 (rw-r--r--), meaning the group and other classes can read it.

This is a deliberate security enforcement built into the SSH client. A private key readable by others is considered compromised — any user on the same system could copy it and impersonate you.

Check the current permissions to confirm:

ls -la ~/.ssh/

You might see something like:

drwxr-xr-x  alice  alice  .ssh/
-rw-r--r--  alice  alice  id_rsa        ← too open (0644)
-rw-r--r--  alice  alice  id_rsa.pub

Solution

Restrict the private key so only the owner can read it:

chmod 600 ~/.ssh/id_rsa

This sets permissions to rw-------: only the owner can read or write the file. Group and other have zero access, satisfying SSH’s requirement.

While you’re there, fix the .ssh directory itself if needed:

chmod 700 ~/.ssh

Then retry:

ssh -i ~/.ssh/id_rsa alice@remote-server.example.com

Why SSH Enforces This

SSH operates on the principle that a private key is a secret credential. If the file is world-readable or group-readable, any process or user sharing the system could read your key and use it to authenticate as you to any server that trusts it.

Unlike a password (which is verified by the remote server), a private key never leaves your machine — the client uses it to sign a challenge. There is no server-side rate limiting or lockout to protect against a stolen key. Once someone has a copy, they have permanent access until you revoke the public key on every remote server. SSH therefore refuses to proceed rather than silently use a key that may already be compromised.


Reference: Correct .ssh Permissions

PathPermissionReason
~/.ssh/700Only owner can enter or list the directory
~/.ssh/id_rsa600Private key — owner read/write only
~/.ssh/id_ed25519600Same rule applies to all private key formats
~/.ssh/id_rsa.pub644Public key — safe to be world-readable
~/.ssh/authorized_keys600SSHd rejects looser permissions on this file too
~/.ssh/known_hosts644Readable by owner; group/other read is acceptable
~/.ssh/config600May contain IdentityFile paths and proxy settings

Apply all of these at once with:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_rsa ~/.ssh/id_ed25519 ~/.ssh/authorized_keys ~/.ssh/config
chmod 644 ~/.ssh/id_rsa.pub ~/.ssh/id_ed25519.pub ~/.ssh/known_hosts

💡 Interview tip: If you are asked why SSH authentication fails even when the key pair is correct, permissions on ~/.ssh/authorized_keys on the remote server are the second most common culprit after the private key itself. SSHd will silently ignore authorized_keys if it is group-writable or world-writable.


Scenario: Nginx Returns 403 Forbidden

The Situation

You have configured Nginx to serve a static site. The config looks correct and the HTML file is present on disk, but every request returns a 403 Forbidden error:

server {
    listen 80;
    server_name example.com;

    root /home/alice/mysite;
    index index.html;
}
curl -I http://example.com/
# HTTP/1.1 403 Forbidden

nginx_403


Diagnosis

A 403 means Nginx can find the location but is denied permission to read it. Because the file visibly exists, the instinct is to check the file itself — but the real cause is almost always further up the directory tree.

Nginx runs as a system user (typically www-data on Debian/Ubuntu, nginx on RHEL/CentOS). To serve /home/alice/mysite/index.html, that user must be able to:

  1. Execute (x) every directory in the path — /home/, /home/alice/, /home/alice/mysite/
  2. Read (r) the file itself — index.html

Failing either check at any level produces a 403. The file’s own permissions are fine to check, but the home directory is the most common culprit.

Check what Nginx is actually blocked on by inspecting the error log:

tail -f /var/log/nginx/error.log

You will see a line like:

[error] 1234#0: *1 open() "/home/alice/mysite/index.html" failed (13: Permission denied)

Now trace the directory permissions from the root down:

ls -la /home/
ls -la /home/alice/
ls -la /home/alice/mysite/
ls -la /home/alice/mysite/index.html

A typical home directory looks like this by default:

drwx------  alice  alice  /home/alice/

700 — only the owner can enter. www-data hits a wall at this directory and never reaches mysite/.


Solution

You need to grant the execute bit on each directory in the path to the user Nginx runs as. There are two approaches:

Option 1 — Add world-execute to the blocking directory (simplest)

chmod o+x /home/alice

This allows any user — including www-data — to traverse into /home/alice. It does not grant read access (they cannot ls the home directory), only traversal.

chmod o+x /home/alice/mysite
chmod o+r /home/alice/mysite/index.html   # if the file itself is also restricted

Option 2 — Add Nginx’s user to the owner’s group (more controlled)

# Add www-data to alice's group
usermod -aG alice www-data

# Grant group execute on the directories
chmod g+x /home/alice
chmod g+x /home/alice/mysite

# Grant group read on the files
chmod -R g+r /home/alice/mysite

Changes to group membership take effect on the next login/session for that user. Reload Nginx after any permission or group change:

systemctl reload nginx

Option 3 — Move the web root outside the home directory (best practice)

Serving files from inside a user’s home directory is the root cause of this class of problem. The standard solution is to host web content under a dedicated directory:

sudo mkdir -p /var/www/mysite
sudo cp -r /home/alice/mysite/* /var/www/mysite/
sudo chown -R www-data:www-data /var/www/mysite
sudo chmod -R 755 /var/www/mysite

Then update the Nginx config:

root /var/www/mysite;

Why Execute on Directories Matters

The read (r) and execute (x) bits mean very different things on directories:

BitOn a directory
rList the contents of the directory (ls)
xTraverse into the directory — required to access anything inside it

A process that lacks x on a directory cannot open, stat, or read any file within it, even if it knows the exact filename. This is why a 403 can occur even when the file itself has world-read permission: the kernel rejects the path traversal before it ever reaches the file.


Reference: Standard Nginx Web Root Permissions

PathOwnerPermissionReason
/var/www/root:root755Base directory, traversable by all
/var/www/mysite/www-data:www-data755Nginx user owns the web root
/var/www/mysite/*.htmlwww-data:www-data644Nginx reads files; no execute needed
/var/www/mysite/uploads/www-data:www-data755If Nginx writes here, owner must match

💡 Interview tip: When debugging a 403, always check the Nginx error log first — it names the exact path and errno. Then walk the directory tree from / down to the file checking for missing x bits on each directory, not just the file itself.


Scenario: Docker Bind Mount — Permission Denied Writing /app/data

The Situation

You have a Dockerized application that writes output files to /app/data. You use a bind mount so the files are accessible on the host:

docker run -v /home/alice/data:/app/data myapp

The container starts, but as soon as the application tries to write a file it crashes with:

PermissionError: [Errno 13] Permission denied: '/app/data/output.csv'

Or, from inside the container:

docker exec -it myapp_container touch /app/data/test
# touch: cannot touch '/app/data/test': Permission denied

The directory /home/alice/data clearly exists on the host and Alice can write to it.


Diagnosis

The key insight is that bind mounts expose the host filesystem directly into the container with no UID translation. The kernel enforces the same ownership and permission rules inside the container as it does on the host — the only thing that changes is the path.

Check what user the container process is running as:

docker exec -it myapp_container id
# uid=1001(appuser) gid=1001(appuser)

Now check who owns the host directory:

ls -la /home/alice/
# drwxr-xr-x  alice  alice  data/
# (alice has uid=1000)

The container process runs as UID 1001 (appuser). The host directory is owned by UID 1000 (alice) with permissions 755 — group and other can only read and traverse, not write. UID 1001 falls into the “other” class and has no write permission.

This is the core mismatch:

Host directory owner: UID 1000  (alice)
Container process:    UID 1001  (appuser)
Directory permission: 755  →  other = r-x  →  no write

You can verify this without even entering the container:

# Find out the UID the container process runs as
docker inspect myapp_container --format '{{.Config.User}}'

# Or check the Dockerfile
grep -i user Dockerfile

Solution

There are four approaches, ordered from quick fix to best practice.

Option 1 — chown the host directory to match the container UID

sudo chown -R 1001:1001 /home/alice/data

The container process (UID 1001) now owns the directory and can write freely. The trade-off: the directory on the host is owned by a UID that may not correspond to a named user on the host system.

Option 2 — Run the container as the host user

Pass --user to make the container process run as the current host user:

docker run --user $(id -u):$(id -g) -v /home/alice/data:/app/data myapp

$(id -u) expands to Alice’s UID (1000) at runtime. The container process now has the same UID as the host directory owner and can write to it. This requires the application inside the container to be able to run as an arbitrary UID (it must not rely on a named user or /etc/passwd entry).

Option 3 — Set the UID in the Dockerfile

If you control the image, align the container user’s UID with the expected host UID at build time:

FROM python:3.12-slim

# Create appuser with UID 1000 to match the host
RUN groupadd -g 1000 appuser && \
    useradd -u 1000 -g appuser -s /bin/sh appuser

WORKDIR /app
COPY . .

USER appuser
# Host directory already owned by UID 1000 (alice) — no chown needed
docker run -v /home/alice/data:/app/data myapp

This is the most portable solution: the image is self-documenting about which UID it expects, and the host directory needs no special preparation beyond normal ownership.

Option 4 — Use a named volume instead of a bind mount

If the application only needs to persist data (not share it with a specific host path), use a Docker-managed named volume:

docker run -v myapp_data:/app/data myapp

Docker creates the volume and sets ownership to the container’s user automatically. There is no host UID involved. Access the data from the host via:

docker run --rm -v myapp_data:/data alpine ls /data

Why UIDs Are Just Numbers

Linux identifies file owners by UID integer, not by username. Usernames are resolved from /etc/passwd only for display. Inside a container, /etc/passwd is the container’s own file, entirely separate from the host’s.

# On the host
id alice
# uid=1000(alice) gid=1000(alice)

# Inside the container — same UID, different name
docker exec myapp_container id 1000
# uid=1000(appuser) gid=1000(appuser)   ← different username, same UID

When the container process (UID 1001) tries to write to a directory owned by UID 1000, the kernel sees two different UIDs and applies the “other” permission bits — regardless of what names appear in either /etc/passwd. The container boundary is irrelevant to the kernel’s permission check; only the UID numbers matter.


Reference: Diagnosing Bind Mount Permission Errors

# 1. Find the UID the container runs as
docker inspect <container> --format '{{.Config.User}}'
docker exec <container> id

# 2. Check host directory ownership
ls -lan /host/path   # -n shows numeric UIDs, bypassing name resolution

# 3. Compare — if UIDs differ, that is the problem
#    owner UID != process UID  →  "other" bits apply

# 4. Check what the kernel sees at mount time
docker exec <container> ls -lan /app/data

💡 Interview tip: Always use ls -ln (numeric UIDs) when debugging cross-container or cross-host permission problems. Username display hides the actual mismatch — two users named appuser on different systems may have completely different UIDs, while two users with different names but the same UID have identical access rights.

Chapter 3: Storage

Interview Questions

This chapter answers to the following questions:


What is LVM and What Are Its Use Cases?

LVM (Logical Volume Manager) is a storage abstraction layer in Linux that sits between raw physical disks and the filesystems that use them. Instead of partitioning a disk directly and formatting it, you pool one or more physical disks (or partitions) under LVM’s control and carve out logical volumes from that pool. The filesystem has no idea how many physical disks are underneath — it just sees a block device of the size you asked for.

LVM introduces three layers of abstraction:

lvm_arch


The Three Layers

Physical Volume (PV)

A Physical Volume is a raw disk or partition that has been initialised for LVM use. LVM writes a small metadata header to it and divides the rest into fixed-size chunks called Physical Extents (PEs) (default 4 MiB each).

pvcreate /dev/sdb              # initialise a disk as a PV
pvcreate /dev/sdc1             # or a partition
pvs                            # list PVs
pvdisplay /dev/sdb             # detailed info

Volume Group (VG)

A Volume Group aggregates one or more PVs into a single storage pool. All PEs from every member PV are available to the group. The VG is the unit of administration — you add capacity by adding a new PV to the VG.

vgcreate vg0 /dev/sdb /dev/sdc   # create VG from two PVs
vgextend vg0 /dev/sdd            # add a third disk later
vgs                              # list VGs
vgdisplay vg0                    # detailed info

Logical Volume (LV)

A Logical Volume is a virtual block device allocated from the VG’s pool of extents. It appears as /dev/vg0/lvname (or /dev/mapper/vg0-lvname), can be formatted with any filesystem, and mounted like any other block device.

lvcreate -L 50G -n data vg0      # create a 50 GiB LV named "data"
lvcreate -l 100%FREE -n backup vg0  # use all remaining space
lvs                              # list LVs
lvdisplay /dev/vg0/data          # detailed info

mkfs.ext4 /dev/vg0/data          # format
mount /dev/vg0/data /mnt/data    # mount

Key Concepts

Linear vs Striped Layouts

By default, LVM writes data linearly: it fills one PV before moving on to the next. You can alternatively create a striped LV that distributes writes across multiple PVs simultaneously, improving throughput at the cost of availability (like RAID 0):

# Striped LV across 3 PVs, 256 KiB stripe size
lvcreate -L 100G -n fastdata -i 3 -I 256 vg0

Use Cases

1. Resize Filesystems Without Downtime

The most common reason to use LVM. With raw partitions, running out of space on /var means repartitioning, which usually requires a reboot and risks data loss. With LVM, you extend a Logical Volume and grow the filesystem in seconds — often while the volume is mounted.

Extend a Logical Volume and its filesystem:

# Step 1: extend the LV by 20 GiB
lvextend -L +20G /dev/vg0/data

# Step 2: grow the filesystem to fill the new space
resize2fs /dev/vg0/data          # ext4
xfs_growfs /mnt/data             # xfs (must be mounted)

Or do both in one command:

lvextend -L +20G --resizefs /dev/vg0/data

Shrink (ext4 only; XFS cannot shrink):

umount /mnt/data
e2fsck -f /dev/vg0/data          # check filesystem first
resize2fs /dev/vg0/data 30G      # shrink filesystem to 30 GiB
lvreduce -L 30G /dev/vg0/data    # shrink LV to match
mount /dev/vg0/data /mnt/data

Always shrink the filesystem first, then the LV. Always grow the LV first, then the filesystem. Reversing the order truncates data.


2. Add Disk Capacity on the Fly

When a server runs low on disk space, you can plug in a new disk and add it to an existing VG — no rebooting, no reformatting, no downtime:

pvcreate /dev/sdd                # prepare new disk
vgextend vg0 /dev/sdd            # add to the pool
lvextend -L +100G --resizefs /dev/vg0/data  # take some of the new space

3. Snapshots for Consistent Backups

LVM can create a snapshot of a Logical Volume — a point-in-time copy that uses copy-on-write (CoW). Only blocks that change after the snapshot is taken are duplicated; unchanged blocks are shared. This means a snapshot of a 100 GiB volume is nearly instant and consumes almost no space initially.

# Create a 5 GiB snapshot of /dev/vg0/data
lvcreate -L 5G -s -n data_snap /dev/vg0/data

# Mount it read-only and back it up
mount -o ro /dev/vg0/data_snap /mnt/snap
rsync -av /mnt/snap/ /backup/data/

# Unmount and remove the snapshot when done
umount /mnt/snap
lvremove /dev/vg0/data_snap

The original volume continues serving live traffic during the backup. If the snapshot fills its allocated space before you remove it, it becomes invalid — size the snapshot generously or use lvextend to grow it if needed.


4. Migrate Data Between Physical Disks

pvmove relocates Physical Extents from one PV to another while the Logical Volume remains mounted. This lets you replace a failing disk or retire old hardware without any downtime:

# Move all data off /dev/sdb to remaining PVs in the VG
pvmove /dev/sdb

# Or move a specific LV's data only
pvmove -n /dev/vg0/data /dev/sdb /dev/sdc

# Once pvmove completes, remove the PV from the VG and decommission it
vgreduce vg0 /dev/sdb
pvremove /dev/sdb

5. Thin Provisioning

Thin provisioning is a storage management technique that allocates physical disk space to applications or virtual machines dynamically as data is written, rather than reserving the entire requested capacity upfront. This approach optimizes storage utilization, reduces hardware costs, and allows administrators to allocate more virtual storage than physically exists.

# Create a thin pool of 200 GiB from the VG
lvcreate -L 200G --thinpool mypool vg0

# Create thin LVs totalling 500 GiB from the 200 GiB pool
lvcreate -V 100G --thin -n vm1 vg0/mypool
lvcreate -V 100G --thin -n vm2 vg0/mypool
lvcreate -V 100G --thin -n vm3 vg0/mypool
lvcreate -V 100G --thin -n vm4 vg0/mypool
lvcreate -V 100G --thin -n vm5 vg0/mypool

Common in VM host environments where VMs are provisioned with generous disk quotas but rarely use their full allocation.


LVM vs Direct Partitioning

FeatureRaw PartitionsLVM
Resize without rebootNoYes (grow always; shrink ext4)
Span multiple disksNo (RAID aside)Yes, transparently
Add disk capacity liveNoYes (vgextend)
Point-in-time snapshotsNoYes (lvcreate -s)
Migrate data between disksRequires rsync + downtimeYes (pvmove)
Thin provisioningNoYes (thin pools)
ComplexityLowMedium
Boot partition (/boot)FineAvoid — most bootloaders don’t support LVM

/boot is typically kept on a plain partition outside LVM because GRUB must read it before the LVM tools are loaded by the initramfs.


Useful Diagnostic Commands

pvs                            # summary of all PVs
vgs                            # summary of all VGs
lvs                            # summary of all LVs

pvdisplay                      # verbose PV info including PE layout
vgdisplay                      # verbose VG info
lvdisplay                      # verbose LV info

lsblk                          # block device tree — shows PV/VG/LV relationships
dmsetup ls --tree              # device-mapper tree (what LVM uses underneath)

vgck vg0                       # check VG metadata consistency

💡 Interview tip: A common interview question is “how would you recover space when a partition is full in production?” The LVM answer is: add a new disk with pvcreate + vgextend, then grow the LV with lvextend --resizefs — all without unmounting or rebooting. Contrast this with the raw-partition answer, which involves repartitioning or moving data, both of which require downtime.