{"count":5,"description":"Notes from things I've built, broken, debugged, and learned along the way. Mostly infrastructure, servers, networking, deployments, and the occasional problem that took longer to solve than it should have.","generated_at":"2026-11-02T13:50:04Z","name":"Posts","posts":[{"canonical_url":"https://ilham.dev/posts/long-running-jobs-should-not-depend-on-my-ssh-connection/","content_markdown":"\nOne of the more annoying parts of working on servers is waiting for something that takes a long time.\n\nA database dump. A restore. Compressing a large directory. Moving data with `rsync`.\n\nYou start the command, watch it for a while, and then your SSH connection drops.\n\nMaybe the Wi-Fi changes. Maybe the VPN reconnects. Maybe the laptop goes to sleep.\n\nThen you SSH back into the server and ask the same question:\n\n**is the process still running?**\n\nI got tired of doing that.\n\n## The problem was not really SSH\n\nSSH disconnecting is normal.\n\nWhat bothered me was that I had made my laptop connection part of a job that might take an hour or more.\n\nFor example:\n\n```bash\npg_dump ...\n```\n\nThe command itself may be completely fine.\n\nBut if I run it directly from a normal SSH shell, I am relying on that shell staying connected for as long as the dump runs.\n\nThat is unnecessary.\n\nFor long-running commands, I would rather have the terminal live on the server.\n\n## There are a few ways to separate the job from my SSH connection\n\nThere is more than one tool for this, and they do not all solve exactly the same problem.\n\nThe ones I keep coming back to are:\n\n- `screen` — persistent terminal session\n- `tmux` — persistent terminal session, similar in purpose to `screen`\n- `nohup` — process keeps running without a terminal to reattach to\n- `systemd` — the operating system manages the process lifecycle\n\nFor the problem that started all of this — a manual dump, restore, transfer, or migration that I may want to come back to — I usually want a terminal I can detach from and reattach to later.\n\nThat puts `screen` and `tmux` in the same category.\n\nI started with `screen` because it already solves that problem with very little ceremony.\n\n## I started with `screen`\n\nFor simple jobs, `screen` solves most of the problem.\n\n```bash\nscreen -S postgres-backup\n```\n\nThen I run the command normally.\n\n```bash\npg_dump ...\n```\n\nIf I want to leave:\n\n```text\nCtrl+A, D\n```\n\nThe terminal keeps running on the server.\n\nLater I can reconnect and attach again.\n\n```bash\nscreen -r postgres-backup\n```\n\nThis is usually enough when I only need one terminal and one command.\n\nThe useful part is not really `screen` itself.\n\nIt is the fact that my SSH connection is no longer responsible for keeping the terminal alive.\n\n## `screen` can keep a log\n\n`screen` is not limited to terminal scrollback.\n\nInside a screen session, I can toggle logging with:\n\n```text\nCtrl+A, H\n```\n\nBy default, this creates a file such as:\n\n```text\nscreenlog.0\n```\n\nThat means I can detach, reconnect later, and still have a record of what happened.\n\nFrom another shell:\n\n```bash\ntail -f screenlog.0\n```\n\nLogging itself is not what separates `screen` from `nohup` or `systemd`.\n\nAll of them can give me some form of log.\n\nThe more important difference is what they manage around the process.\n\n## `screen` is not limited to one terminal\n\nBefore comparing it with `tmux`, one thing is worth making clear: `screen` can already manage multiple windows and split the display.\n\nInside an existing `screen` session, I can create another window with:\n\n```text\nCtrl+A, C\n```\n\nThen move between windows:\n\n```text\nCtrl+A, N next window\nCtrl+A, P previous window\nCtrl+A, \" list windows\n```\n\nSo one session can look like this:\n\n```text\nscreen session: maintenance\n\n├── window 0: pg_dump\n├── window 1: htop\n├── window 2: tail -f backup.log\n└── window 3: df -h\n```\n\n`screen` can also split the display into regions.\n\nHorizontal split:\n\n```text\nCtrl+A, S\n```\n\nVertical split:\n\n```text\nCtrl+A, |\n```\n\nAnd I can move focus between regions with:\n\n```text\nCtrl+A, Tab\n```\n\nSo I can have something roughly like:\n\n```text\n┌────────────────────────────┐\n│ pg_dump │\n├──────────────┬─────────────┤\n│ htop │ tail -f log │\n└──────────────┴─────────────┘\n```\n\nThis matters because the difference between `screen` and `tmux` is not that one supports multiple terminals and the other does not.\n\nBoth do.\n\nThe terminology is slightly different:\n\n```text\nscreen tmux\n\nsession session\nwindow window\nregion pane\nCtrl+A Ctrl+B\n```\n\nFor the kind of long-running jobs I usually do on one server, their core capabilities overlap a lot.\n\n## `screen` and `tmux` are very similar for this problem\n\nFor the problem in this post, I do not think there is a strong winner between `screen` and `tmux`.\n\nBoth let me create a terminal session on the server, detach from it, lose my SSH connection, reconnect later, and return to the same terminal.\n\nWith `tmux`, the basic flow looks like this:\n\n```bash\ntmux new -s postgres-backup\n```\n\nDetach:\n\n```text\nCtrl+B, D\n```\n\nThen later:\n\n```bash\ntmux attach -t postgres-backup\n```\n\nSo if I am running a database dump, restore, `rsync`, compression job, or manual migration and my SSH connection disappears, either tool solves the main problem.\n\n`screen` can have multiple windows and split regions.\n\n`tmux` can have multiple windows and panes.\n\nFor this use case, choosing between them is mostly about which interface, key bindings, and workflow I prefer.\n\nBoth tools are lightweight. `screen` tends to have a slightly smaller footprint, but for the kind of servers I work on, the difference is small enough that I would choose based on availability and preference rather than memory usage.\n\n`tmux` does have features that are different from `screen`. One example is synchronized panes, where the same keyboard input can be sent to several panes at once. That can be useful when several panes contain SSH sessions to different servers and I want to run the same inspection command everywhere.\n\nThat is a real difference, but it is not something I need for the problem that started this article.\n\nIf all I want is for a two-hour process to keep running when my SSH connection drops, synchronized panes do not help me.\n\nA persistent terminal does.\n\nSo I do not choose `tmux` because `screen` is missing some basic capability here. I am happy to use either one depending on what is already installed and which interface I feel like using.\n\nThe more important limitation is shared by both tools: keeping the terminal alive does not tell me whether the process inside it is still healthy.\n\n## But staying connected is only half of the problem\n\nThere is another failure mode.\n\nThe SSH connection survives.\n\nThe terminal survives.\n\nBut the process itself dies.\n\nMaybe the disk becomes full.\n\nMaybe the database connection disappears.\n\nMaybe the process gets killed because the server runs out of memory.\n\nMaybe the command simply exits with an error.\n\nA `screen` or `tmux` session can still be there even though the actual job stopped twenty minutes ago.\n\nThat means:\n\n\u003e keeping a terminal alive is not the same as monitoring the process inside it.\n\nThis distinction matters.\n\n## I want output I can inspect later\n\nFor longer jobs, I usually prefer having a log instead of relying entirely on terminal scrollback.\n\nA pattern I often use is:\n\n```bash\npg_dump ... 2\u003e\u00261 | tee backup.log\n```\n\nThere are a few shell concepts in that command that are worth understanding.\n\n## What `|` does\n\nThe pipe operator:\n\n```bash\n|\n```\n\ntakes the output from the command on the left and sends it to the command on the right.\n\nSo:\n\n```bash\npg_dump ... | tee backup.log\n```\n\nlooks roughly like this:\n\n```text\npg_dump\n |\n v\n tee\n |\n +--\u003e terminal\n |\n +--\u003e backup.log\n```\n\nWithout `tee`, I could redirect the output directly into a file:\n\n```bash\npg_dump ... \u003e backup.log\n```\n\nThat works.\n\nBut now I am no longer seeing the same output live in the terminal.\n\nFor a long-running job I am actively watching, I usually want both.\n\n## Why `tee`?\n\n`tee` lets me keep the output visible while also writing it to a file.\n\n```bash\npg_dump ... | tee backup.log\n```\n\nSo I get:\n\n```text\nterminal\n+\nbackup.log\n```\n\nIf the job fails later, I do not have to depend on terminal scrollback.\n\nI can inspect the log directly.\n\n```bash\ntail -n 100 backup.log\n```\n\nor follow it while the process is still running:\n\n```bash\ntail -f backup.log\n```\n\n## What `2\u003e\u00261` means\n\nLinux commands normally have two output streams that matter here.\n\n```text\nstdout = 1\nstderr = 2\n```\n\n`stdout` is normal output.\n\n`stderr` is usually used for errors and warnings.\n\nWhen I write:\n\n```bash\n2\u003e\u00261\n```\n\nI am telling the shell to send stderr to the same place as stdout.\n\nSo:\n\n```bash\npg_dump ... 2\u003e\u00261 | tee backup.log\n```\n\nmeans both normal output and errors go through `tee`.\n\n```text\n stdout ----\\\npg_dump +--\u003e tee --\u003e terminal\n stderr ----/ \\\n --\u003e backup.log\n```\n\nWithout `2\u003e\u00261`, some errors may still appear in the terminal but not end up inside `backup.log`.\n\nFor troubleshooting, I usually want both streams together.\n\n## Why I also use `pipefail`\n\nThere is one subtle problem with:\n\n```bash\npg_dump ... 2\u003e\u00261 | tee backup.log\n```\n\nThis is now a pipeline containing two commands:\n\n```text\npg_dump\ntee\n```\n\nBy default, the shell may return the exit status of the last command in the pipeline.\n\nThat means `pg_dump` could fail while `tee` successfully writes the error into the log.\n\nThat is not the result I want.\n\nSo before running it, I use:\n\n```bash\nset -o pipefail\n```\n\nThen:\n\n```bash\nset -o pipefail\npg_dump ... 2\u003e\u00261 | tee backup.log\n```\n\nAfter it finishes:\n\n```bash\necho $?\n```\n\nA result of:\n\n```text\n0\n```\n\nmeans success.\n\nAnything else means something in the pipeline failed and I should inspect the log.\n\nFor a manual long-running job, this gives me a useful combination:\n\n```text\nlive output\npersistent log\nstderr in the same log\nuseful exit status\n```\n\n## But `screen` can log, so why use anything else?\n\nThis is the more interesting question.\n\nSuppose I run a database dump inside `screen`.\n\nI can have:\n\n```text\nscreen session\n+\nscreenlog.0\n+\nthe command itself\n```\n\nIf SSH disconnects, the session stays alive.\n\nIf I reconnect, I can attach again.\n\nIf logging is enabled, I can inspect what happened.\n\nFor an ad-hoc database dump, that may genuinely be all I need.\n\nThe difference appears when I ask another question:\n\n**who knows whether the process actually succeeded?**\n\n`screen` mostly manages the terminal session.\n\nThe log is evidence of what happened.\n\nBut the log is not the same thing as the machine managing the state of the job.\n\n## What happens when the process suddenly stops?\n\nImagine the dump fails because the disk becomes full.\n\nInside a normal interactive `screen` session, the structure is roughly:\n\n```text\nscreen\n |\n +--\u003e shell\n |\n +--\u003e pg_dump\n```\n\n`pg_dump` exits.\n\nBut the shell is still running.\n\nSo when I reconnect, I may simply find:\n\n```text\npg_dump: error ...\nbisku@server:~$\n```\n\nThat is actually useful.\n\nThe session survived.\n\nThe error may still be visible.\n\nAnd if screen logging was enabled, it may also be inside:\n\n```text\nscreenlog.0\n```\n\nThe same idea applies to `tmux`.\n\nThe terminal can survive even though the process inside it has already stopped.\n\n## Sometimes I only want to know if it is still doing work\n\nFor a manual job I am actively watching, sophisticated monitoring is often unnecessary.\n\nI can check whether the process still exists:\n\n```bash\npgrep -af pg_dump\n```\n\nI prefer this over:\n\n```bash\nps aux | grep pg_dump\n```\n\nfor quick checks because I do not have to deal with the `grep` process appearing in the result.\n\nIf the dump writes to a file, I can also see whether it is still growing.\n\n```bash\nwatch -n 2 'du -h backup.dump'\n```\n\nAnd I can check available disk space.\n\n```bash\nwatch -n 2 'df -h'\n```\n\nOr watch the log:\n\n```bash\ntail -f backup.log\n```\n\nThese are simple signals, but together they answer the question I normally care about:\n\n**is useful work still happening?**\n\n## Process alive does not always mean process healthy\n\nA process appearing in:\n\n```bash\npgrep -af pg_dump\n```\n\nonly tells me that the process exists.\n\nIt could still be waiting on I/O.\n\nIt could be stuck waiting on another service.\n\nThe output file could have stopped growing.\n\nThat is why, for a manual operation, I usually look at several things instead of relying on one signal.\n\n```text\nprocess exists\noutput keeps growing\nlog keeps changing\ndisk still has space\n```\n\nThat is usually enough for something I am actively watching.\n\n## `nohup` solves a smaller problem\n\nSometimes I do not care about coming back to the terminal.\n\nI only want the process to survive logout.\n\nThat is where `nohup` is useful.\n\n```bash\nnohup rsync ... \u003e rsync.log 2\u003e\u00261 \u0026\n```\n\nThere are three parts here.\n\n`nohup` makes the process ignore the hangup signal normally associated with its terminal going away.\n\nThis:\n\n```bash\n\u003e rsync.log 2\u003e\u00261\n```\n\nstores stdout and stderr in the log.\n\nAnd:\n\n```bash\n\u0026\n```\n\nruns the process in the background.\n\nThen I can disconnect.\n\nLater:\n\n```bash\ntail -f rsync.log\n```\n\nThe important difference is that there is no terminal session to reattach to.\n\nWith `screen` or `tmux`, I keep the terminal context.\n\nWith `nohup`, I mostly keep the process.\n\nFor manual operations where I expect to come back and inspect things, I usually prefer `screen` or `tmux`.\n\n## Stopping the job is another thing I want to control\n\nThere is another practical difference I care about.\n\nSometimes the job is not failing.\n\nI am the one who wants to stop it.\n\nMaybe I started the wrong database dump.\n\nMaybe the destination path is wrong.\n\nMaybe disk usage is growing faster than expected.\n\nIn that situation, I prefer stopping the actual command cleanly instead of immediately killing the environment around it.\n\n## Stopping something inside `screen`\n\nIf the dump is running inside `screen`, I attach first:\n\n```bash\nscreen -r postgres-backup\n```\n\nThen stop the command normally:\n\n```text\nCtrl+C\n```\n\nThat sends an interrupt to the foreground process.\n\nAfter the command has stopped, I can inspect the output, check the result, and then close the shell:\n\n```bash\nexit\n```\n\nIf I really want to terminate the entire screen session from outside:\n\n```bash\nscreen -S postgres-backup -X quit\n```\n\nBut that is not usually my first choice.\n\nI would rather stop the job intentionally, inspect what happened, and then close the session.\n\n## Stopping something inside `tmux`\n\nI use the same idea with `tmux`.\n\nAttach:\n\n```bash\ntmux attach -t postgres-backup\n```\n\nThen:\n\n```text\nCtrl+C\n```\n\nIf I am done with the shell:\n\n```bash\nexit\n```\n\nIf I want to remove the entire session from outside:\n\n```bash\ntmux kill-session -t postgres-backup\n```\n\n`tmux` also lets me be more granular.\n\nI can kill one pane:\n\n```bash\ntmux kill-pane\n```\n\nor one window:\n\n```bash\ntmux kill-window\n```\n\nThat is useful if the session contains several things and I do not want to destroy the whole workspace.\n\n## Stopping a process started with `nohup`\n\n`nohup` is different because there is no terminal session to return to.\n\nI need to find the process.\n\nFor example:\n\n```bash\npgrep -af pg_dump\n```\n\nSuppose I see:\n\n```text\n18432 pg_dump -Fc database_name -f backup.dump\n```\n\nThen I can ask the process to stop:\n\n```bash\nkill 18432\n```\n\n`kill` without another option sends `SIGTERM`.\n\nThat gives the process a chance to shut down normally.\n\nThen I check again:\n\n```bash\npgrep -af pg_dump\n```\n\nIf nothing comes back, the process has stopped.\n\nThis is one reason I find `nohup` less comfortable for manual work.\n\nOnce I leave the original shell, I am managing the process by PID instead of returning to the same terminal.\n\n## Then there is `systemd`\n\nAt some point, a job stops feeling like something I should manually babysit.\n\nMaybe the same backup needs to run every night.\n\nMaybe an application must always be running.\n\nMaybe I want the server itself to remember whether a job succeeded or failed.\n\nThat is where `systemd` comes in.\n\n`systemd` is the system and service manager used by many modern Linux distributions. As the system instance, it normally starts as PID 1 during boot and manages userspace services for the machine.\n\nIts job is much broader than keeping a terminal open.\n\nIt can manage things such as:\n\n- services;\n- background processes;\n- startup dependencies;\n- timers;\n- process status;\n- restart policies;\n- process lifecycle.\n\nInstead of me opening a terminal and running:\n\n```bash\n/home/bisku/backup-postgres.sh\n```\n\nI can describe that job to `systemd` as a unit.\n\nA command or service is commonly represented by a `.service` unit, for example:\n\n```text\nbackup-postgres.service\n```\n\nConceptually, that tells the server:\n\n```text\nthis is the command\nthis is how it should run\nthis is what happened when it stopped\n```\n\nNow the process belongs to the server's service manager instead of one of my terminal sessions.\n\nFor scheduled work, `systemd` also has timer units.\n\nFor example:\n\n```text\nbackup-postgres.timer\n```\n\ncan trigger:\n\n```text\nbackup-postgres.service\n```\n\nat a defined schedule.\n\nThat is why I see `systemd` as a different category from `screen`, `tmux`, or `nohup`.\n\n`screen` gives me a persistent terminal.\n\n`systemd` gives the process a lifecycle managed by the operating system.\n\n## `systemd` knows how the job ended\n\nBecause `systemd` launched and tracks the unit, I can ask it for the current state:\n\n```bash\nsystemctl status backup-postgres\n```\n\nIf something failed, I can see an explicit failed state and inspect the exit result instead of only trying to infer success from terminal output.\n\nI can also inspect the unit's journal:\n\n```bash\njournalctl -u backup-postgres\n```\n\nor follow it live:\n\n```bash\njournalctl -fu backup-postgres\n```\n\nAgain, the advantage is not simply that `systemd` has logs.\n\n`screen` can log too.\n\n`nohup` can redirect output to a file too.\n\nThe useful part is that `systemd` understands the relationship between the process and its lifecycle: when it started, whether it is still running, and how it exited.\n\nFor a service that is supposed to stay running, it can also have a restart policy such as:\n\n```ini\nRestart=on-failure\n```\n\nThat is process supervision.\n\nIt is something fundamentally different from keeping a shell alive in `screen`.\n\nFor a one-shot database dump, I usually would not blindly restart it. For an API, worker, or another long-running service, automatic restart may make much more sense.\n\n## Stopping something managed by `systemd`\n\nWith `systemd`, the process has an explicit lifecycle.\n\nSo stopping it is straightforward:\n\n```bash\nsudo systemctl stop backup-postgres\n```\n\nThen:\n\n```bash\nsystemctl status backup-postgres\n```\n\nThe intent is clear.\n\nI am not looking for a PID.\n\nI am not finding the correct terminal.\n\nI am telling the service manager that this unit should stop.\n\nThat distinction becomes increasingly useful as a job becomes more important.\n\n## I prefer graceful stop before killing the session\n\nFor manual work, my rule is simple:\n\n\u003e stop the process first, then remove the environment around it.\n\nFor `screen`:\n\n```text\nattach\nCtrl+C\ninspect\nexit\n```\n\nFor `tmux`:\n\n```text\nattach\nCtrl+C\ninspect\nexit\n```\n\nI only reach for:\n\n```bash\nscreen -S postgres-backup -X quit\n```\n\nor:\n\n```bash\ntmux kill-session -t postgres-backup\n```\n\nwhen I actually want to remove the session itself.\n\nThe job and the terminal around the job are two different things.\n\n## Logs and state are not the same thing\n\nThis is the distinction that finally made these tools easier for me to reason about.\n\n```text\nscreen\n persistent terminal session\n + multiple windows and split regions\n + optional log\n\ntmux\n persistent terminal session\n + multiple windows and panes\n + optional logging\n + additional workspace features\n\nnohup\n detached process\n + redirected log\n\nsystemd\n managed process\n + status\n + exit result\n + journal\n + scheduling\n + restart policy\n```\n\nAll of them can leave behind text describing what happened.\n\nBut logs answer:\n\n**what did the process print?**\n\nProcess management answers:\n\n**what happened to the process?**\n\nThose are different questions.\n\n## And even `systemd` is not monitoring me\n\nSuppose `systemd` correctly records:\n\n```text\nbackup-postgres.service = failed\n```\n\nThat still does not mean I know about it.\n\nThe server knows.\n\nI do not necessarily know.\n\nSo the progression becomes:\n\n```text\nWill the terminal survive?\n |\n v\nscreen / tmux\n\nWill the process survive logout?\n |\n v\nscreen / tmux / nohup\n\nWill the machine understand the process state?\n |\n v\nsystemd\n\nWill I know when something fails?\n |\n v\nmonitoring / alerting\n```\n\nThat is why I do not think of these tools as direct replacements for each other.\n\nThey solve different layers of the same operational problem.\n\n## What I use now\n\nMy rule is fairly simple.\n\nFor a manual long-running job where I want to return to the same terminal, I am comfortable with either:\n\n```text\nscreen / tmux\n```\n\nThat covers most of the cases that originally made me look for these tools: database dumps, restores, large transfers, compression, and manual migrations.\n\nFor this problem, I do not need to treat `tmux` as an upgrade from `screen`. Both solve the part I care about: the terminal stays on the server even when my SSH connection disappears.\n\nWhich one I use can simply come down to preference, familiarity, or what is already installed.\n\n`tmux` has extra workspace features such as synchronized panes, but those are useful for different workflows. They are not necessary just to protect a long-running command from an SSH disconnect.\n\nFor something I want to start in the background and mostly forget about:\n\n```text\nnohup\n```\n\nFor something recurring or important enough that the machine needs to understand whether it succeeded:\n\n```text\nsystemd\n```\n\nAnd if failure cannot be silent:\n\n```text\nmonitoring / alerting\n```\n\nThe thing I originally wanted to fix was simple.\n\nI was tired of waiting for a long-running process and then suddenly losing my SSH connection.\n\nThat led me to `screen` and `tmux`.\n\nThen I realized there was another problem.\n\nA terminal can survive while the process inside it has already failed.\n\nAnd then another one.\n\nThe server can know that a process failed while I still know nothing about it.\n\nSo now I separate those concerns.\n\nMy laptop should not determine whether a two-hour database dump survives.\n\nMy terminal should not be the only place where failure exists.\n\nAnd when I do need to stop something, I want to stop the job deliberately, not just kill whatever terminal happens to be holding it.\n\n## References\n\n- [GNU Screen: windows](https://www.gnu.org/software/screen/manual/html_node/Windows.html)\n- [GNU Screen: regions](https://www.gnu.org/software/screen/manual/html_node/Regions.html)\n- [GNU Screen: logging](https://www.gnu.org/software/screen/manual/html_node/Log.html)\n- [GNU Screen: command reference](https://www.gnu.org/software/screen/manual/screen.html)\n- [tmux manual](https://man.openbsd.org/tmux)\n- [systemd system and service manager](https://www.freedesktop.org/software/systemd/man/latest/systemd.html)\n- [journalctl](https://www.freedesktop.org/software/systemd/man/latest/journalctl.html)","content_text":"One of the more annoying parts of working on servers is waiting for something that takes a long time.\nA database dump. A restore. Compressing a large directory. Moving data with rsync.\nYou start the command, watch it for a while, and then your SSH connection drops.\nMaybe the Wi-Fi changes. Maybe the VPN reconnects. Maybe the laptop goes to sleep.\nThen you SSH back into the server and ask the same question:\nis the process still running?\nI got tired of doing that.\nThe problem was not really SSH SSH disconnecting is normal.\nWhat bothered me was that I had made my laptop connection part of a job that might take an hour or more.\nFor example:\npg_dump ... The command itself may be completely fine.\nBut if I run it directly from a normal SSH shell, I am relying on that shell staying connected for as long as the dump runs.\nThat is unnecessary.\nFor long-running commands, I would rather have the terminal live on the server.\nThere are a few ways to separate the job from my SSH connection There is more than one tool for this, and they do not all solve exactly the same problem.\nThe ones I keep coming back to are:\nscreen — persistent terminal session tmux — persistent terminal session, similar in purpose to screen nohup — process keeps running without a terminal to reattach to systemd — the operating system manages the process lifecycle For the problem that started all of this — a manual dump, restore, transfer, or migration that I may want to come back to — I usually want a terminal I can detach from and reattach to later.\nThat puts screen and tmux in the same category.\nI started with screen because it already solves that problem with very little ceremony.\nI started with screen For simple jobs, screen solves most of the problem.\nscreen -S postgres-backup Then I run the command normally.\npg_dump ... If I want to leave:\nCtrl+A, D The terminal keeps running on the server.\nLater I can reconnect and attach again.\nscreen -r postgres-backup This is usually enough when I only need one terminal and one command.\nThe useful part is not really screen itself.\nIt is the fact that my SSH connection is no longer responsible for keeping the terminal alive.\nscreen can keep a log screen is not limited to terminal scrollback.\nInside a screen session, I can toggle logging with:\nCtrl+A, H By default, this creates a file such as:\nscreenlog.0 That means I can detach, reconnect later, and still have a record of what happened.\nFrom another shell:\ntail -f screenlog.0 Logging itself is not what separates screen from nohup or systemd.\nAll of them can give me some form of log.\nThe more important difference is what they manage around the process.\nscreen is not limited to one terminal Before comparing it with tmux, one thing is worth making clear: screen can already manage multiple windows and split the display.\nInside an existing screen session, I can create another window with:\nCtrl+A, C Then move between windows:\nCtrl+A, N next window Ctrl+A, P previous window Ctrl+A, \" list windows So one session can look like this:\nscreen session: maintenance ├── window 0: pg_dump ├── window 1: htop ├── window 2: tail -f backup.log └── window 3: df -h screen can also split the display into regions.\nHorizontal split:\nCtrl+A, S Vertical split:\nCtrl+A, | And I can move focus between regions with:\nCtrl+A, Tab So I can have something roughly like:\n┌────────────────────────────┐ │ pg_dump │ ├──────────────┬─────────────┤ │ htop │ tail -f log │ └──────────────┴─────────────┘ This matters because the difference between screen and tmux is not that one supports multiple terminals and the other does not.\nBoth do.\nThe terminology is slightly different:\nscreen tmux session session window window region pane Ctrl+A Ctrl+B For the kind of long-running jobs I usually do on one server, their core capabilities overlap a lot.\nscreen and tmux are very similar for this problem For the problem in this post, I do not think there is a strong winner between screen and tmux.\nBoth let me create a terminal session on the server, detach from it, lose my SSH connection, reconnect later, and return to the same terminal.\nWith tmux, the basic flow looks like this:\ntmux new -s postgres-backup Detach:\nCtrl+B, D Then later:\ntmux attach -t postgres-backup So if I am running a database dump, restore, rsync, compression job, or manual migration and my SSH connection disappears, either tool solves the main problem.\nscreen can have multiple windows and split regions.\ntmux can have multiple windows and panes.\nFor this use case, choosing between them is mostly about which interface, key bindings, and workflow I prefer.\nBoth tools are lightweight. screen tends to have a slightly smaller footprint, but for the kind of servers I work on, the difference is small enough that I would choose based on availability and preference rather than memory usage.\ntmux does have features that are different from screen. One example is synchronized panes, where the same keyboard input can be sent to several panes at once. That can be useful when several panes contain SSH sessions to different servers and I want to run the same inspection command everywhere.\nThat is a real difference, but it is not something I need for the problem that started this article.\nIf all I want is for a two-hour process to keep running when my SSH connection drops, synchronized panes do not help me.\nA persistent terminal does.\nSo I do not choose tmux because screen is missing some basic capability here. I am happy to use either one depending on what is already installed and which interface I feel like using.\nThe more important limitation is shared by both tools: keeping the terminal alive does not tell me whether the process inside it is still healthy.\nBut staying connected is only half of the problem There is another failure mode.\nThe SSH connection survives.\nThe terminal survives.\nBut the process itself dies.\nMaybe the disk becomes full.\nMaybe the database connection disappears.\nMaybe the process gets killed because the server runs out of memory.\nMaybe the command simply exits with an error.\nA screen or tmux session can still be there even though the actual job stopped twenty minutes ago.\nThat means:\nkeeping a terminal alive is not the same as monitoring the process inside it.\nThis distinction matters.\nI want output I can inspect later For longer jobs, I usually prefer having a log instead of relying entirely on terminal scrollback.\nA pattern I often use is:\npg_dump ... 2\u003e\u00261 | tee backup.log There are a few shell concepts in that command that are worth understanding.\nWhat | does The pipe operator:\n| takes the output from the command on the left and sends it to the command on the right.\nSo:\npg_dump ... | tee backup.log looks roughly like this:\npg_dump | v tee | +--\u003e terminal | +--\u003e backup.log Without tee, I could redirect the output directly into a file:\npg_dump ... \u003e backup.log That works.\nBut now I am no longer seeing the same output live in the terminal.\nFor a long-running job I am actively watching, I usually want both.\nWhy tee? tee lets me keep the output visible while also writing it to a file.\npg_dump ... | tee backup.log So I get:\nterminal + backup.log If the job fails later, I do not have to depend on terminal scrollback.\nI can inspect the log directly.\ntail -n 100 backup.log or follow it while the process is still running:\ntail -f backup.log What 2\u003e\u00261 means Linux commands normally have two output streams that matter here.\nstdout = 1 stderr = 2 stdout is normal output.\nstderr is usually used for errors and warnings.\nWhen I write:\n2\u003e\u00261 I am telling the shell to send stderr to the same place as stdout.\nSo:\npg_dump ... 2\u003e\u00261 | tee backup.log means both normal output and errors go through tee.\nstdout ----\\ pg_dump +--\u003e tee --\u003e terminal stderr ----/ \\ --\u003e backup.log Without 2\u003e\u00261, some errors may still appear in the terminal but not end up inside backup.log.\nFor troubleshooting, I usually want both streams together.\nWhy I also use pipefail There is one subtle problem with:\npg_dump ... 2\u003e\u00261 | tee backup.log This is now a pipeline containing two commands:\npg_dump tee By default, the shell may return the exit status of the last command in the pipeline.\nThat means pg_dump could fail while tee successfully writes the error into the log.\nThat is not the result I want.\nSo before running it, I use:\nset -o pipefail Then:\nset -o pipefail pg_dump ... 2\u003e\u00261 | tee backup.log After it finishes:\necho $? A result of:\n0 means success.\nAnything else means something in the pipeline failed and I should inspect the log.\nFor a manual long-running job, this gives me a useful combination:\nlive output persistent log stderr in the same log useful exit status But screen can log, so why use anything else? This is the more interesting question.\nSuppose I run a database dump inside screen.\nI can have:\nscreen session + screenlog.0 + the command itself If SSH disconnects, the session stays alive.\nIf I reconnect, I can attach again.\nIf logging is enabled, I can inspect what happened.\nFor an ad-hoc database dump, that may genuinely be all I need.\nThe difference appears when I ask another question:\nwho knows whether the process actually succeeded?\nscreen mostly manages the terminal session.\nThe log is evidence of what happened.\nBut the log is not the same thing as the machine managing the state of the job.\nWhat happens when the process suddenly stops? Imagine the dump fails because the disk becomes full.\nInside a normal interactive screen session, the structure is roughly:\nscreen | +--\u003e shell | +--\u003e pg_dump pg_dump exits.\nBut the shell is still running.\nSo when I reconnect, I may simply find:\npg_dump: error ... bisku@server:~$ That is actually useful.\nThe session survived.\nThe error may still be visible.\nAnd if screen logging was enabled, it may also be inside:\nscreenlog.0 The same idea applies to tmux.\nThe terminal can survive even though the process inside it has already stopped.\nSometimes I only want to know if it is still doing work For a manual job I am actively watching, sophisticated monitoring is often unnecessary.\nI can check whether the process still exists:\npgrep -af pg_dump I prefer this over:\nps aux | grep pg_dump for quick checks because I do not have to deal with the grep process appearing in the result.\nIf the dump writes to a file, I can also see whether it is still growing.\nwatch -n 2 'du -h backup.dump' And I can check available disk space.\nwatch -n 2 'df -h' Or watch the log:\ntail -f backup.log These are simple signals, but together they answer the question I normally care about:\nis useful work still happening?\nProcess alive does not always mean process healthy A process appearing in:\npgrep -af pg_dump only tells me that the process exists.\nIt could still be waiting on I/O.\nIt could be stuck waiting on another service.\nThe output file could have stopped growing.\nThat is why, for a manual operation, I usually look at several things instead of relying on one signal.\nprocess exists output keeps growing log keeps changing disk still has space That is usually enough for something I am actively watching.\nnohup solves a smaller problem Sometimes I do not care about coming back to the terminal.\nI only want the process to survive logout.\nThat is where nohup is useful.\nnohup rsync ... \u003e rsync.log 2\u003e\u00261 \u0026 There are three parts here.\nnohup makes the process ignore the hangup signal normally associated with its terminal going away.\nThis:\n\u003e rsync.log 2\u003e\u00261 stores stdout and stderr in the log.\nAnd:\n\u0026 runs the process in the background.\nThen I can disconnect.\nLater:\ntail -f rsync.log The important difference is that there is no terminal session to reattach to.\nWith screen or tmux, I keep the terminal context.\nWith nohup, I mostly keep the process.\nFor manual operations where I expect to come back and inspect things, I usually prefer screen or tmux.\nStopping the job is another thing I want to control There is another practical difference I care about.\nSometimes the job is not failing.\nI am the one who wants to stop it.\nMaybe I started the wrong database dump.\nMaybe the destination path is wrong.\nMaybe disk usage is growing faster than expected.\nIn that situation, I prefer stopping the actual command cleanly instead of immediately killing the environment around it.\nStopping something inside screen If the dump is running inside screen, I attach first:\nscreen -r postgres-backup Then stop the command normally:\nCtrl+C That sends an interrupt to the foreground process.\nAfter the command has stopped, I can inspect the output, check the result, and then close the shell:\nexit If I really want to terminate the entire screen session from outside:\nscreen -S postgres-backup -X quit But that is not usually my first choice.\nI would rather stop the job intentionally, inspect what happened, and then close the session.\nStopping something inside tmux I use the same idea with tmux.\nAttach:\ntmux attach -t postgres-backup Then:\nCtrl+C If I am done with the shell:\nexit If I want to remove the entire session from outside:\ntmux kill-session -t postgres-backup tmux also lets me be more granular.\nI can kill one pane:\ntmux kill-pane or one window:\ntmux kill-window That is useful if the session contains several things and I do not want to destroy the whole workspace.\nStopping a process started with nohup nohup is different because there is no terminal session to return to.\nI need to find the process.\nFor example:\npgrep -af pg_dump Suppose I see:\n18432 pg_dump -Fc database_name -f backup.dump Then I can ask the process to stop:\nkill 18432 kill without another option sends SIGTERM.\nThat gives the process a chance to shut down normally.\nThen I check again:\npgrep -af pg_dump If nothing comes back, the process has stopped.\nThis is one reason I find nohup less comfortable for manual work.\nOnce I leave the original shell, I am managing the process by PID instead of returning to the same terminal.\nThen there is systemd At some point, a job stops feeling like something I should manually babysit.\nMaybe the same backup needs to run every night.\nMaybe an application must always be running.\nMaybe I want the server itself to remember whether a job succeeded or failed.\nThat is where systemd comes in.\nsystemd is the system and service manager used by many modern Linux distributions. As the system instance, it normally starts as PID 1 during boot and manages userspace services for the machine.\nIts job is much broader than keeping a terminal open.\nIt can manage things such as:\nservices; background processes; startup dependencies; timers; process status; restart policies; process lifecycle. Instead of me opening a terminal and running:\n/home/bisku/backup-postgres.sh I can describe that job to systemd as a unit.\nA command or service is commonly represented by a .service unit, for example:\nbackup-postgres.service Conceptually, that tells the server:\nthis is the command this is how it should run this is what happened when it stopped Now the process belongs to the server’s service manager instead of one of my terminal sessions.\nFor scheduled work, systemd also has timer units.\nFor example:\nbackup-postgres.timer can trigger:\nbackup-postgres.service at a defined schedule.\nThat is why I see systemd as a different category from screen, tmux, or nohup.\nscreen gives me a persistent terminal.\nsystemd gives the process a lifecycle managed by the operating system.\nsystemd knows how the job ended Because systemd launched and tracks the unit, I can ask it for the current state:\nsystemctl status backup-postgres If something failed, I can see an explicit failed state and inspect the exit result instead of only trying to infer success from terminal output.\nI can also inspect the unit’s journal:\njournalctl -u backup-postgres or follow it live:\njournalctl -fu backup-postgres Again, the advantage is not simply that systemd has logs.\nscreen can log too.\nnohup can redirect output to a file too.\nThe useful part is that systemd understands the relationship between the process and its lifecycle: when it started, whether it is still running, and how it exited.\nFor a service that is supposed to stay running, it can also have a restart policy such as:\nRestart=on-failure That is process supervision.\nIt is something fundamentally different from keeping a shell alive in screen.\nFor a one-shot database dump, I usually would not blindly restart it. For an API, worker, or another long-running service, automatic restart may make much more sense.\nStopping something managed by systemd With systemd, the process has an explicit lifecycle.\nSo stopping it is straightforward:\nsudo systemctl stop backup-postgres Then:\nsystemctl status backup-postgres The intent is clear.\nI am not looking for a PID.\nI am not finding the correct terminal.\nI am telling the service manager that this unit should stop.\nThat distinction becomes increasingly useful as a job becomes more important.\nI prefer graceful stop before killing the session For manual work, my rule is simple:\nstop the process first, then remove the environment around it.\nFor screen:\nattach Ctrl+C inspect exit For tmux:\nattach Ctrl+C inspect exit I only reach for:\nscreen -S postgres-backup -X quit or:\ntmux kill-session -t postgres-backup when I actually want to remove the session itself.\nThe job and the terminal around the job are two different things.\nLogs and state are not the same thing This is the distinction that finally made these tools easier for me to reason about.\nscreen persistent terminal session + multiple windows and split regions + optional log tmux persistent terminal session + multiple windows and panes + optional logging + additional workspace features nohup detached process + redirected log systemd managed process + status + exit result + journal + scheduling + restart policy All of them can leave behind text describing what happened.\nBut logs answer:\nwhat did the process print?\nProcess management answers:\nwhat happened to the process?\nThose are different questions.\nAnd even systemd is not monitoring me Suppose systemd correctly records:\nbackup-postgres.service = failed That still does not mean I know about it.\nThe server knows.\nI do not necessarily know.\nSo the progression becomes:\nWill the terminal survive? | v screen / tmux Will the process survive logout? | v screen / tmux / nohup Will the machine understand the process state? | v systemd Will I know when something fails? | v monitoring / alerting That is why I do not think of these tools as direct replacements for each other.\nThey solve different layers of the same operational problem.\nWhat I use now My rule is fairly simple.\nFor a manual long-running job where I want to return to the same terminal, I am comfortable with either:\nscreen / tmux That covers most of the cases that originally made me look for these tools: database dumps, restores, large transfers, compression, and manual migrations.\nFor this problem, I do not need to treat tmux as an upgrade from screen. Both solve the part I care about: the terminal stays on the server even when my SSH connection disappears.\nWhich one I use can simply come down to preference, familiarity, or what is already installed.\ntmux has extra workspace features such as synchronized panes, but those are useful for different workflows. They are not necessary just to protect a long-running command from an SSH disconnect.\nFor something I want to start in the background and mostly forget about:\nnohup For something recurring or important enough that the machine needs to understand whether it succeeded:\nsystemd And if failure cannot be silent:\nmonitoring / alerting The thing I originally wanted to fix was simple.\nI was tired of waiting for a long-running process and then suddenly losing my SSH connection.\nThat led me to screen and tmux.\nThen I realized there was another problem.\nA terminal can survive while the process inside it has already failed.\nAnd then another one.\nThe server can know that a process failed while I still know nothing about it.\nSo now I separate those concerns.\nMy laptop should not determine whether a two-hour database dump survives.\nMy terminal should not be the only place where failure exists.\nAnd when I do need to stop something, I want to stop the job deliberately, not just kill whatever terminal happens to be holding it.\nReferences GNU Screen: windows GNU Screen: regions GNU Screen: logging GNU Screen: command reference tmux manual systemd system and service manager journalctl ","date":"2026-10-30","description":"I got tired of long-running jobs depending on an SSH session, so I started separating terminal persistence, logging, process state, and monitoring.","id":"long-running-jobs-should-not-depend-on-my-ssh-connection","markdown":"https://ilham.dev/posts/long-running-jobs-should-not-depend-on-my-ssh-connection/index.md","markdown_url":"https://ilham.dev/posts/long-running-jobs-should-not-depend-on-my-ssh-connection/index.md","reading_minutes":17,"summary":"One of the more annoying parts of working on servers is waiting for something that takes a long time.\nA database dump. A restore. Compressing a large directory. Moving data with rsync.","tags":["devops","linux","automation"],"title":"Long-running jobs should not depend on my SSH connection","url":"https://ilham.dev/posts/long-running-jobs-should-not-depend-on-my-ssh-connection/","word_count":3487},{"canonical_url":"https://ilham.dev/posts/when-df-h-is-full-why-i-usually-extend-storage-with-lvm/","content_markdown":"\nMost storage expansion problems I run into start with something simple:\n\n```bash\ndf -h\n```\n\nImagine I see this:\n\n```text\nFilesystem Size Used Avail Use% Mounted on\n/dev/mapper/ubuntu--vg-root--lv 50G 20G 30G 40% /\n/dev/sda2 2G 250M 1.7G 13% /boot\n/dev/mapper/ubuntu--vg-home--lv 100G 95G 5G 95% /home\n```\n\n`/home` is almost full.\n\nMaybe it contains application files, deployment directories, uploads, source code, user files, or other data that has grown over time.\n\nThe path is already in use. Everything is working. I just need more space.\n\nWhat I want is simple:\n\n```text\n/home\n100 GB\n```\n\nbecomes:\n\n```text\n/home\n300 GB\n```\n\nwithout changing `/home` into another path.\n\nThat is the problem this article is about.\n\nSometimes this is easy without LVM.\n\nSometimes the partition layout gets in the way.\n\nAnd sometimes the easiest solution is to attach another disk or iSCSI LUN and let LVM add that capacity underneath the same `/home` filesystem.\n\n# First: what does `df -h` actually show?\n\n`df` means **disk free**.\n\nThe `-h` option means **human-readable**.\n\nSo:\n\n```bash\ndf -h\n```\n\nshows filesystem capacity in values such as:\n\n```text\n100G\n95G\n5G\n```\n\nThe important part is this:\n\n```text\n/home\n```\n\nis only the **mount point**.\n\nIt is not necessarily the physical disk.\n\nThere may be several storage layers underneath it.\n\nFor example, this:\n\n```text\n/dev/mapper/ubuntu--vg-home--lv → /home\n```\n\nalready tells me `/home` is backed by an LVM Logical Volume.\n\n# Disk, partition, filesystem, and mount point are different\n\nA simple non-LVM storage stack might look like:\n\n```text\nDisk\n ↓\nPartition\n ↓\nFilesystem\n ↓\nMount point\n```\n\nFor example:\n\n```text\n/dev/sda\n ↓\n/dev/sda3\n ↓\next4\n ↓\n/home\n```\n\nThese are different things:\n\n```text\n/dev/sda\n→ whole disk\n\n/dev/sda3\n→ one partition on that disk\n\next4\n→ filesystem\n\n/home\n→ directory where the filesystem is mounted\n```\n\nThe user or application normally only cares about:\n\n```text\n/home\n```\n\n# What is `/dev`?\n\nLinux exposes devices through special device nodes under:\n\n```text\n/dev\n```\n\nFor example:\n\n```text\n/dev/sda\n/dev/sdb\n/dev/sdc\n```\n\nare block-device names.\n\n`/dev/sda` usually represents a whole disk-like device.\n\nA number after it normally means a partition:\n\n```text\n/dev/sda\n├── /dev/sda1\n├── /dev/sda2\n├── /dev/sda3\n└── /dev/sda4\n```\n\nNVMe devices look different:\n\n```text\n/dev/nvme0n1\n/dev/nvme0n1p1\n```\n\nVirtio disks may appear as:\n\n```text\n/dev/vda\n/dev/vdb\n```\n\nSo I never assume the new disk will definitely be `/dev/sdb`.\n\nI check:\n\n```bash\nlsblk\n```\n\n# The real problem: `/home` is full\n\nSuppose I do **not** use LVM.\n\nMy server might look like this:\n\n```text\n/dev/sda\n├── /dev/sda1 → EFI\n├── /dev/sda2 → /boot\n├── /dev/sda3 → /\n└── /dev/sda4 → /home\n```\n\nAnd:\n\n```bash\ndf -h /home\n```\n\nshows:\n\n```text\nFilesystem Size Used Avail Use% Mounted on\n/dev/sda4 100G 95G 5G 95% /home\n```\n\nThe requirement is not:\n\n\u003e Give me another directory.\n\nIt is:\n\n\u003e Make the existing `/home` larger.\n\n# Non-LVM: when growing `/home` is easy\n\nSuppose `/home` is the final partition on the disk:\n\n```text\n/dev/sda\n├── sda1 → EFI\n├── sda2 → /boot\n├── sda3 → /\n└── sda4 → /home 100 GB\n```\n\nThe virtual disk is 200 GB.\n\nI increase it in Proxmox to 400 GB.\n\nLinux may now see:\n\n```text\n/dev/sda = 400 GB\n```\n\nwhile `/dev/sda4` is still only 100 GB.\n\nThe layout is now conceptually:\n\n```text\n[/ EFI ][ /boot ][ / ][ /home 100G ][ free 200G ]\n```\n\nThis is the easy case.\n\nThe free space is directly after `/home`.\n\nSo `/home` can expand into it.\n\nFor an ext4 filesystem, a common flow is conceptually:\n\n```bash\nlsblk\ngrowpart /dev/sda 4\nresize2fs /dev/sda4\ndf -h /home\n```\n\nFor XFS, after the partition is enlarged, the filesystem is normally grown from the mount point:\n\n```bash\nxfs_growfs /home\n```\n\nIn this situation, non-LVM is perfectly fine.\n\nLVM is not required just for the sake of using LVM.\n\n# Why a normal partition needs free space next to it\n\nA normal partition represents one continuous range of sectors on one disk.\n\nThink of the disk as a row of rooms:\n\n```text\n[ room 1 ][ room 2 ][ room 3 ][ empty ]\n```\n\nIf `/home` is the last room before the empty space:\n\n```text\n[/home][free]\n```\n\nit can simply grow to the right:\n\n```text\n[/home................]\n```\n\nBut this is different:\n\n```text\n[/home][another partition][free]\n```\n\n`/home` cannot jump over that other partition.\n\nThe free capacity exists, but it is in the wrong place.\n\n# A real non-LVM case where adding disk capacity does not immediately help `/home`\n\nSuppose the original disk is:\n\n```text\n/dev/sda 200 GB\n├── sda1 1 GB → EFI\n├── sda2 2 GB → /boot\n├── sda3 80 GB → /\n├── sda4 100 GB → /home\n└── sda5 10 GB → swap\n```\n\nIgnore the exact totals here; the important part is the order:\n\n```text\n[ EFI ][ /boot ][ / ][ /home ][ swap ]\n```\n\nNow `/home` is almost full.\n\nI enlarge the virtual disk by 200 GB.\n\nThe disk becomes larger, but the layout becomes:\n\n```text\n[ EFI ][ /boot ][ / ][ /home ][ swap ][ free 200G ]\n```\n\nThe new space exists.\n\nBut `/home` cannot directly use it.\n\nWhy?\n\nBecause swap is sitting between `/home` and the free space.\n\nThe partition that can naturally grow into the new free space is actually the **last partition**, which in this example is swap:\n\n```text\n[ /home ][ swap ][ free ]\n```\n\ncould become:\n\n```text\n[ /home ][ swap............. ]\n```\n\nBut that does not solve the problem because the thing that is full is `/home`, not swap.\n\n# Can I just make swap smaller and give that space to `/home`?\n\nNot quite.\n\nSuppose I start with:\n\n```text\n[/home 100G][swap 10G][free 200G]\n```\n\nIf I only shrink swap from its right side:\n\n```text\n[/home 100G][swap 5G][free 205G]\n```\n\nswap is still in the middle.\n\n`/home` still cannot reach the free space.\n\nWhat I really need is to move swap out of the way.\n\nBecause swap does not contain permanent files, this particular case is relatively manageable.\n\nConceptually:\n\n```text\nBefore:\n[/home 100G][swap 10G][free 200G]\n\nTemporarily remove/recreate swap:\n[/home 100G][free 210G]\n\nGrow /home:\n[/home 300G][free 10G]\n\nCreate swap again at the end:\n[/home 300G][swap 10G]\n```\n\nSo non-LVM is **not impossible** here.\n\nIt simply requires changing the partition layout.\n\n# If this is urgent, what do I do?\n\nBefore touching partitions, I first identify exactly what is mounted where:\n\n```bash\nlsblk -f\nfindmnt /home\nswapon --show\nfree -h\n```\n\nThat matters because deleting the wrong partition is obviously much worse than running out of disk space.\n\nIf the only thing blocking `/home` is swap, the high-level emergency procedure is:\n\n```text\nverify the layout\n ↓\nmake sure RAM can survive without the current swap\n ↓\ndisable the old swap\n ↓\nremove/recreate the swap partition layout\n ↓\ngrow the /home partition\n ↓\ngrow the filesystem\n ↓\ncreate swap again at the end\n ↓\nenable swap again\n```\n\nIf swap is heavily used and RAM is tight, I would not blindly run `swapoff`.\n\nA temporary swap file can be useful before changing the old swap partition.\n\nThe important lesson is that an urgent non-LVM fix may be possible, but it can involve more moving parts than simply adding capacity.\n\n# What if the blocking partition is `/home`, ordinary data, or system data?\n\nThe answer depends on what that partition contains.\n\n## If the blocking partition is swap\n\nThis is usually the easiest case because swap can be recreated.\n\n```text\n[target][swap][free]\n```\n\ncan often be rearranged into:\n\n```text\n[target........][swap]\n```\n\n## If the blocking partition contains movable data\n\nSuppose the layout is:\n\n```text\n[/home][/archive][free]\n```\n\nor:\n\n```text\n[/home][/srv][free]\n```\n\nIf the blocking filesystem contains ordinary data, I can move that data somewhere else first.\n\nConceptually:\n\n```text\n1. copy the blocking filesystem to another disk\n2. verify the copy\n3. unmount the old filesystem\n4. remove or recreate its old partition\n5. grow /home\n6. recreate the other filesystem if needed\n7. restore the data\n```\n\nWhen moving Linux files, I care about more than file contents.\n\nI also want to preserve ownership, permissions, timestamps, symlinks, ACLs, and extended attributes.\n\nA command such as:\n\n```bash\nrsync -aHAX /source/ /mnt/temporary-disk/\n```\n\nis often more appropriate than a simple file copy.\n\nI still verify the result before removing the original filesystem.\n\nSometimes the cleaner design is to leave that data permanently on the new disk instead of putting it back.\n\n## If the blocking partition contains system data\n\nNow the risk is much higher.\n\nFor example:\n\n```text\n[/home][/var][free]\n```\n\nor:\n\n```text\n[/home][/usr][free]\n```\n\nor a layout involving the root filesystem:\n\n```text\n[/home][/][free]\n```\n\n`/var`, `/usr`, and `/` are not just ordinary folders that I casually move while Linux is running.\n\nThey may contain or be actively used by:\n\n```text\nsystem services\nlogs\npackage-manager data\nDocker\nlibraries\ndatabases\napplication state\nboot-time files\n```\n\nFor a case like that, I usually prefer one of these approaches:\n\n```text\nattach another disk and migrate /home there\n```\n\nor, if the partition table really must be rearranged:\n\n```text\nuse a maintenance or rescue environment\n```\n\nThe more important the blocking filesystem is to the running OS, the less attractive live partition surgery becomes.\n\nA simple way to think about it is:\n\n| Blocking partition | Relative difficulty |\n| --- | --- |\n| swap | easiest to recreate |\n| archive / backup data | usually easy to move |\n| `/home` or another user-data filesystem | usually movable with planning |\n| application data | movable, but may require downtime |\n| `/var` | sensitive |\n| `/usr` | very sensitive |\n| `/` | very sensitive |\n| boot / EFI | avoid touching unless necessary |\n\n# Another non-LVM problem: the new capacity arrives as another disk\n\nThere is another common situation.\n\nInstead of enlarging the existing disk, Proxmox or the storage platform gives me another disk:\n\n```text\n/dev/sda = existing disk\n/dev/sdb = new 200 GB disk\n```\n\nMy `/home` filesystem may still live on:\n\n```text\n/dev/sda4\n ↓\next4\n ↓\n/home\n```\n\nA normal ext4 or XFS filesystem on `/dev/sda4` cannot simply say:\n\n```text\nuse the old blocks from /dev/sda4\nplus 200 GB from /dev/sdb\n```\n\nThey are separate block devices.\n\nWithout another storage layer, I normally need to migrate `/home` to the new disk, create another mount layout, or otherwise redesign the storage.\n\nThis is one of the reasons LVM exists.\n\n# Why LVM exists\n\nTraditional partitions are closely tied to **where storage physically sits**.\n\nLVM adds an abstraction layer so I can manage storage based more on **how much capacity I need** than on exactly where that capacity happens to be located.\n\nThe simplest summary is:\n\n\u003e **Partitions manage pieces of disks. LVM manages pools of storage.**\n\nWithout LVM, I think about things such as:\n\n```text\nWhich partition is last?\nWhere is the free space?\nWhat partition is in the way?\nCan I enlarge this exact disk?\nDo I need to move another filesystem first?\n```\n\nWith LVM, I can create a storage pool and allocate logical volumes from it.\n\nThe main terms are:\n\n```text\nPV\nPhysical Volume\n\nVG\nVolume Group\n\nLV\nLogical Volume\n```\n\nI think of them like this:\n\n```text\nPV\n→ storage given to LVM\n\nVG\n→ a pool containing that storage\n\nLV\n→ a logical block device allocated from the pool\n```\n\nThe stack becomes:\n\n```text\nDisk or LUN\n ↓\n PV\n ↓\n VG\n ↓\n LV\n ↓\nFilesystem\n ↓\n /home\n```\n\n# A simple `/home` example with LVM\n\nSuppose `/home` starts with 100 GB:\n\n```text\n/dev/sda → PV\n ↓\n ubuntu-vg\n ↓\n home-lv 100 GB\n ↓\n ext4\n ↓\n /home\n```\n\nNow `/home` reaches 95 GB used.\n\nI need another 200 GB.\n\nInstead of rearranging old partitions, I attach another 200 GB disk:\n\n```text\n/dev/sdb = 200 GB\n```\n\nI can give that disk to LVM:\n\n```text\nold PV 100 GB ─┐\n ├── ubuntu-vg = 300 GB total capacity\nnew PV 200 GB ─┘\n```\n\nThen the Logical Volume backing `/home` can grow:\n\n```text\nhome-lv\n100 GB → 300 GB\n```\n\nThe filesystem grows too:\n\n```text\nfilesystem\n100 GB → 300 GB\n```\n\nAnd the user still sees:\n\n```text\n/home\n```\n\nNothing about the application path needs to change.\n\n# How can two different disks look like one storage device?\n\nThis is the part that can feel strange at first.\n\nSuppose I have:\n\n```text\n/dev/sda = 100 GB\n/dev/sdb = 200 GB\n```\n\nThey are still physically two separate block devices.\n\nLVM does not magically weld them into one physical disk.\n\nInstead, LVM creates a **mapping layer**.\n\nLVM divides Physical Volumes into chunks called **Physical Extents (PEs)**.\n\nLogical Volumes are built from corresponding **Logical Extents (LEs)**.\n\nConceptually:\n\n```text\nPhysical storage:\n\nDisk A\n[ PE1 ][ PE2 ][ PE3 ]\n\nDisk B\n[ PE4 ][ PE5 ][ PE6 ]\n```\n\nLVM can map them into one logical sequence:\n\n```text\nLogical Volume:\n[ LE1 ][ LE2 ][ LE3 ][ LE4 ][ LE5 ][ LE6 ]\n```\n\nFor example:\n\n```text\nLE1 → PE on Disk A\nLE2 → PE on Disk A\nLE3 → PE on Disk A\nLE4 → PE on Disk B\nLE5 → PE on Disk B\nLE6 → PE on Disk B\n```\n\nThe filesystem above LVM sees one logical block device:\n\n```text\n/dev/ubuntu-vg/home-lv\n```\n\nIt does not need to care that some blocks ultimately live on one PV and some on another.\n\nThat is the trick.\n\n# LVM does not physically insert one disk into another\n\nLVM is **not** doing this:\n\n```text\nDisk A\n[ old data ][ Disk B somehow inserted here ][ more data ]\n```\n\nThe physical devices remain separate.\n\nLVM simply maintains a logical map above them:\n\n```text\nDisk A ─┐\n ├── VG → LV → filesystem → /home\nDisk B ─┘\n```\n\nThis is why the physical space does not need to be adjacent in the same way a normal partition does.\n\n# LVM does not magically fix a blocked partition\n\nThere is an important limitation.\n\nSuppose the PV itself is a partition:\n\n```text\n/dev/sda\n├── /dev/sda3 → PV\n└── /dev/sda4 → another filesystem\n```\n\nand free space exists after `/dev/sda4`.\n\nLVM cannot make `/dev/sda3` jump over `/dev/sda4`.\n\nThe normal partition rule still applies.\n\nThe advantage is that I may not need to grow `/dev/sda3` at all.\n\nI can attach another disk:\n\n```text\n/dev/sdb → new PV\n```\n\nand add it to the existing VG:\n\n```text\n/dev/sda3 → old PV ─┐\n ├── VG → LV → /home\n/dev/sdb → new PV ─┘\n```\n\nSo LVM does not remove the physical partition limitation.\n\nIt gives me another way around it.\n\n# A real LVM expansion flow\n\nSuppose:\n\n```bash\ndf -h /home\n```\n\nshows:\n\n```text\nFilesystem Size Used Avail Use% Mounted on\n/dev/mapper/ubuntu--vg-home--lv 100G 95G 5G 95% /home\n```\n\nI inspect the storage layers:\n\n```bash\nlsblk -f\npvs\nvgs\nlvs\n```\n\nSuppose the VG has no free capacity and I attach a new empty 200 GB disk that Linux identifies as `/dev/sdb`.\n\nAfter carefully verifying that `/dev/sdb` is the intended empty disk, the LVM part is conceptually:\n\n```bash\npvcreate /dev/sdb\nvgextend ubuntu-vg /dev/sdb\nlvextend -l +100%FREE /dev/ubuntu-vg/home-lv\n```\n\nThen I grow the filesystem.\n\nFor ext4:\n\n```bash\nresize2fs /dev/ubuntu-vg/home-lv\n```\n\nFor XFS:\n\n```bash\nxfs_growfs /home\n```\n\nAfterward:\n\n```bash\ndf -h /home\n```\n\nmay show something like:\n\n```text\nFilesystem Size Used Avail Use% Mounted on\n/dev/mapper/ubuntu--vg-home--lv 300G 95G 205G 32% /home\n```\n\nThe mount point is still exactly the same:\n\n```text\n/home\n```\n\nOnly the storage layers underneath it changed.\n\n# What if I enlarge the existing LVM disk instead?\n\nLVM also works when I enlarge an existing virtual disk.\n\nIf the whole device is already a PV:\n\n```text\n/dev/sdb\n ↓\n PV\n```\n\nand Proxmox changes it from:\n\n```text\n100 GB\n```\n\nto:\n\n```text\n300 GB\n```\n\nthen, after Linux sees the larger device, LVM can discover the new capacity with:\n\n```bash\npvresize /dev/sdb\n```\n\nThen the flow is:\n\n```text\nlarger PV\n ↓\nVG gains free capacity\n ↓\nLV grows\n ↓\nfilesystem grows\n ↓\n/home grows\n```\n\nIf the PV is inside a partition such as `/dev/sdb1`, that partition may still need to grow first.\n\nAgain, LVM does not make partition geometry disappear.\n\n# Why `df -h` can stay unchanged after I add storage\n\nAdding capacity at one layer does not automatically resize every layer above it.\n\nFor example:\n\n```text\nProxmox disk\n300 GB\n```\n\ncan still have:\n\n```text\npartition\n100 GB\n```\n\nor:\n\n```text\nPV\n100 GB\n```\n\nor:\n\n```text\nLV\n100 GB\n```\n\nor:\n\n```text\nfilesystem\n100 GB\n```\n\n`df -h` reports the filesystem's usable capacity.\n\nSo I think from bottom to top:\n\n```text\nProxmox / iSCSI\n ↓\nblock device\n ↓\npartition, if used\n ↓\nPV, if LVM is used\n ↓\nVG\n ↓\nLV\n ↓\nfilesystem\n ↓\n/home\n```\n\nThen I ask:\n\n\u003e Which layer has not grown yet?\n\n# My usual troubleshooting flow\n\nI start with:\n\n```bash\ndf -h\n```\n\nThen:\n\n```bash\nlsblk -f\n```\n\nFor `/home` specifically:\n\n```bash\nfindmnt /home\ndf -T /home\n```\n\nIf LVM is involved:\n\n```bash\npvs\nvgs\nlvs\n```\n\nThose commands answer different questions:\n\n```text\ndf -h\n→ which filesystem is running out of space?\n\nlsblk -f\n→ what disks, partitions, filesystems, and mounts exist?\n\nfindmnt /home\n→ what device actually backs /home?\n\npvs\n→ which devices are LVM Physical Volumes?\n\nvgs\n→ how much total and free capacity exists in the Volume Group?\n\nlvs\n→ how large is the Logical Volume?\n```\n\n# Non-LVM versus LVM\n\n| | Non-LVM | LVM |\n| --- | --- | --- |\n| Keep `/home` as the same path | Yes | Yes |\n| Grow an existing disk | Easy when the layout permits | Easy when the layout permits |\n| Partition needs adjacent free space | Yes | Yes, if growing that same PV partition |\n| Add another disk to the same logical storage | Not directly | Yes |\n| Combine capacity from multiple disks/LUNs | Not directly | Yes, through PVs and a VG |\n| Avoid moving an unrelated blocking partition | Often requires migration or rearrangement | Often possible by adding another PV |\n| Number of storage layers | Fewer | More |\n| Troubleshooting | Simpler | Requires PV/VG/LV knowledge |\n| Simple fixed-purpose server | Excellent | Sometimes unnecessary |\n| Long-lived server that keeps growing | Can become layout-dependent | Usually more flexible |\n\n# When I am happy with non-LVM\n\nNon-LVM is perfectly reasonable when the layout is simple.\n\nFor example:\n\n```text\n/dev/sda4\n ↓\next4\n ↓\n/home\n```\n\nIf `/dev/sda4` is the final partition and the virtual disk can be enlarged, growing `/home` can be very straightforward.\n\nThat is a good design.\n\n# When I prefer LVM\n\nI prefer LVM when I expect storage requirements to change over the lifetime of the server.\n\nA server may start with:\n\n```text\n/home = 100 GB\n```\n\nthen need:\n\n```text\n/home = 300 GB\n```\n\nand later:\n\n```text\n/home = 500 GB\n```\n\nThe next capacity increase may come from:\n\n```text\na larger Proxmox virtual disk\nanother Proxmox virtual disk\nan iSCSI LUN\nanother iSCSI LUN\n```\n\nLVM gives me a layer where those physical storage changes do not have to map directly to changes in the path the application or user sees.\n\n# LVM is not RAID\n\nCombining capacity through LVM does not automatically make the data redundant.\n\nIf I have:\n\n```text\n/dev/sda ─┐\n ├── VG\n/dev/sdb ─┘\n```\n\nthat does not mean every block exists on both devices.\n\nIt only means both devices can contribute storage capacity.\n\nIn simple terms:\n\n```text\nLVM\n→ flexible storage management\n\nRAID\n→ disk layout and possible redundancy\n\nBackup\n→ recovery from data loss\n```\n\nThey solve different problems.\n\n# The point\n\nThe problem I care about is not simply:\n\n\u003e How do I attach another disk?\n\nIt is:\n\n\u003e `/home` is almost full. How do I make the same `/home` larger with the least disruptive change?\n\nWithout LVM, that may be very easy when `/home` is the last partition and the free space is directly after it.\n\nBut if the layout is:\n\n```text\n[/home][swap][free]\n```\n\nor:\n\n```text\n[/home][another filesystem][free]\n```\n\nthen the partition in the middle has to be dealt with first.\n\nIf it is swap, it can usually be recreated.\n\nIf it contains movable data, I can migrate that data first.\n\nIf it contains important system data, I may prefer migration, maintenance mode, or a different disk instead of live partition surgery.\n\nLVM exists to reduce how much storage management depends on that physical partition layout.\n\nIt lets me think in terms of a pool:\n\n```text\nDisk / LUN 1 → PV ─┐\n ├── VG → LV → filesystem → /home\nDisk / LUN 2 → PV ─┘\n```\n\nThe underlying devices can change while `/home` remains `/home`.\n\nThat is the practical reason I usually prefer LVM for Linux servers that I expect to keep growing.\n","content_text":"Most storage expansion problems I run into start with something simple:\ndf -h Imagine I see this:\nFilesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-root--lv 50G 20G 30G 40% / /dev/sda2 2G 250M 1.7G 13% /boot /dev/mapper/ubuntu--vg-home--lv 100G 95G 5G 95% /home /home is almost full.\nMaybe it contains application files, deployment directories, uploads, source code, user files, or other data that has grown over time.\nThe path is already in use. Everything is working. I just need more space.\nWhat I want is simple:\n/home 100 GB becomes:\n/home 300 GB without changing /home into another path.\nThat is the problem this article is about.\nSometimes this is easy without LVM.\nSometimes the partition layout gets in the way.\nAnd sometimes the easiest solution is to attach another disk or iSCSI LUN and let LVM add that capacity underneath the same /home filesystem.\nFirst: what does df -h actually show? df means disk free.\nThe -h option means human-readable.\nSo:\ndf -h shows filesystem capacity in values such as:\n100G 95G 5G The important part is this:\n/home is only the mount point.\nIt is not necessarily the physical disk.\nThere may be several storage layers underneath it.\nFor example, this:\n/dev/mapper/ubuntu--vg-home--lv → /home already tells me /home is backed by an LVM Logical Volume.\nDisk, partition, filesystem, and mount point are different A simple non-LVM storage stack might look like:\nDisk ↓ Partition ↓ Filesystem ↓ Mount point For example:\n/dev/sda ↓ /dev/sda3 ↓ ext4 ↓ /home These are different things:\n/dev/sda → whole disk /dev/sda3 → one partition on that disk ext4 → filesystem /home → directory where the filesystem is mounted The user or application normally only cares about:\n/home What is /dev? Linux exposes devices through special device nodes under:\n/dev For example:\n/dev/sda /dev/sdb /dev/sdc are block-device names.\n/dev/sda usually represents a whole disk-like device.\nA number after it normally means a partition:\n/dev/sda ├── /dev/sda1 ├── /dev/sda2 ├── /dev/sda3 └── /dev/sda4 NVMe devices look different:\n/dev/nvme0n1 /dev/nvme0n1p1 Virtio disks may appear as:\n/dev/vda /dev/vdb So I never assume the new disk will definitely be /dev/sdb.\nI check:\nlsblk The real problem: /home is full Suppose I do not use LVM.\nMy server might look like this:\n/dev/sda ├── /dev/sda1 → EFI ├── /dev/sda2 → /boot ├── /dev/sda3 → / └── /dev/sda4 → /home And:\ndf -h /home shows:\nFilesystem Size Used Avail Use% Mounted on /dev/sda4 100G 95G 5G 95% /home The requirement is not:\nGive me another directory.\nIt is:\nMake the existing /home larger.\nNon-LVM: when growing /home is easy Suppose /home is the final partition on the disk:\n/dev/sda ├── sda1 → EFI ├── sda2 → /boot ├── sda3 → / └── sda4 → /home 100 GB The virtual disk is 200 GB.\nI increase it in Proxmox to 400 GB.\nLinux may now see:\n/dev/sda = 400 GB while /dev/sda4 is still only 100 GB.\nThe layout is now conceptually:\n[/ EFI ][ /boot ][ / ][ /home 100G ][ free 200G ] This is the easy case.\nThe free space is directly after /home.\nSo /home can expand into it.\nFor an ext4 filesystem, a common flow is conceptually:\nlsblk growpart /dev/sda 4 resize2fs /dev/sda4 df -h /home For XFS, after the partition is enlarged, the filesystem is normally grown from the mount point:\nxfs_growfs /home In this situation, non-LVM is perfectly fine.\nLVM is not required just for the sake of using LVM.\nWhy a normal partition needs free space next to it A normal partition represents one continuous range of sectors on one disk.\nThink of the disk as a row of rooms:\n[ room 1 ][ room 2 ][ room 3 ][ empty ] If /home is the last room before the empty space:\n[/home][free] it can simply grow to the right:\n[/home................] But this is different:\n[/home][another partition][free] /home cannot jump over that other partition.\nThe free capacity exists, but it is in the wrong place.\nA real non-LVM case where adding disk capacity does not immediately help /home Suppose the original disk is:\n/dev/sda 200 GB ├── sda1 1 GB → EFI ├── sda2 2 GB → /boot ├── sda3 80 GB → / ├── sda4 100 GB → /home └── sda5 10 GB → swap Ignore the exact totals here; the important part is the order:\n[ EFI ][ /boot ][ / ][ /home ][ swap ] Now /home is almost full.\nI enlarge the virtual disk by 200 GB.\nThe disk becomes larger, but the layout becomes:\n[ EFI ][ /boot ][ / ][ /home ][ swap ][ free 200G ] The new space exists.\nBut /home cannot directly use it.\nWhy?\nBecause swap is sitting between /home and the free space.\nThe partition that can naturally grow into the new free space is actually the last partition, which in this example is swap:\n[ /home ][ swap ][ free ] could become:\n[ /home ][ swap............. ] But that does not solve the problem because the thing that is full is /home, not swap.\nCan I just make swap smaller and give that space to /home? Not quite.\nSuppose I start with:\n[/home 100G][swap 10G][free 200G] If I only shrink swap from its right side:\n[/home 100G][swap 5G][free 205G] swap is still in the middle.\n/home still cannot reach the free space.\nWhat I really need is to move swap out of the way.\nBecause swap does not contain permanent files, this particular case is relatively manageable.\nConceptually:\nBefore: [/home 100G][swap 10G][free 200G] Temporarily remove/recreate swap: [/home 100G][free 210G] Grow /home: [/home 300G][free 10G] Create swap again at the end: [/home 300G][swap 10G] So non-LVM is not impossible here.\nIt simply requires changing the partition layout.\nIf this is urgent, what do I do? Before touching partitions, I first identify exactly what is mounted where:\nlsblk -f findmnt /home swapon --show free -h That matters because deleting the wrong partition is obviously much worse than running out of disk space.\nIf the only thing blocking /home is swap, the high-level emergency procedure is:\nverify the layout ↓ make sure RAM can survive without the current swap ↓ disable the old swap ↓ remove/recreate the swap partition layout ↓ grow the /home partition ↓ grow the filesystem ↓ create swap again at the end ↓ enable swap again If swap is heavily used and RAM is tight, I would not blindly run swapoff.\nA temporary swap file can be useful before changing the old swap partition.\nThe important lesson is that an urgent non-LVM fix may be possible, but it can involve more moving parts than simply adding capacity.\nWhat if the blocking partition is /home, ordinary data, or system data? The answer depends on what that partition contains.\nIf the blocking partition is swap This is usually the easiest case because swap can be recreated.\n[target][swap][free] can often be rearranged into:\n[target........][swap] If the blocking partition contains movable data Suppose the layout is:\n[/home][/archive][free] or:\n[/home][/srv][free] If the blocking filesystem contains ordinary data, I can move that data somewhere else first.\nConceptually:\n1. copy the blocking filesystem to another disk 2. verify the copy 3. unmount the old filesystem 4. remove or recreate its old partition 5. grow /home 6. recreate the other filesystem if needed 7. restore the data When moving Linux files, I care about more than file contents.\nI also want to preserve ownership, permissions, timestamps, symlinks, ACLs, and extended attributes.\nA command such as:\nrsync -aHAX /source/ /mnt/temporary-disk/ is often more appropriate than a simple file copy.\nI still verify the result before removing the original filesystem.\nSometimes the cleaner design is to leave that data permanently on the new disk instead of putting it back.\nIf the blocking partition contains system data Now the risk is much higher.\nFor example:\n[/home][/var][free] or:\n[/home][/usr][free] or a layout involving the root filesystem:\n[/home][/][free] /var, /usr, and / are not just ordinary folders that I casually move while Linux is running.\nThey may contain or be actively used by:\nsystem services logs package-manager data Docker libraries databases application state boot-time files For a case like that, I usually prefer one of these approaches:\nattach another disk and migrate /home there or, if the partition table really must be rearranged:\nuse a maintenance or rescue environment The more important the blocking filesystem is to the running OS, the less attractive live partition surgery becomes.\nA simple way to think about it is:\nBlocking partition Relative difficulty swap easiest to recreate archive / backup data usually easy to move /home or another user-data filesystem usually movable with planning application data movable, but may require downtime /var sensitive /usr very sensitive / very sensitive boot / EFI avoid touching unless necessary Another non-LVM problem: the new capacity arrives as another disk There is another common situation.\nInstead of enlarging the existing disk, Proxmox or the storage platform gives me another disk:\n/dev/sda = existing disk /dev/sdb = new 200 GB disk My /home filesystem may still live on:\n/dev/sda4 ↓ ext4 ↓ /home A normal ext4 or XFS filesystem on /dev/sda4 cannot simply say:\nuse the old blocks from /dev/sda4 plus 200 GB from /dev/sdb They are separate block devices.\nWithout another storage layer, I normally need to migrate /home to the new disk, create another mount layout, or otherwise redesign the storage.\nThis is one of the reasons LVM exists.\nWhy LVM exists Traditional partitions are closely tied to where storage physically sits.\nLVM adds an abstraction layer so I can manage storage based more on how much capacity I need than on exactly where that capacity happens to be located.\nThe simplest summary is:\nPartitions manage pieces of disks. LVM manages pools of storage.\nWithout LVM, I think about things such as:\nWhich partition is last? Where is the free space? What partition is in the way? Can I enlarge this exact disk? Do I need to move another filesystem first? With LVM, I can create a storage pool and allocate logical volumes from it.\nThe main terms are:\nPV Physical Volume VG Volume Group LV Logical Volume I think of them like this:\nPV → storage given to LVM VG → a pool containing that storage LV → a logical block device allocated from the pool The stack becomes:\nDisk or LUN ↓ PV ↓ VG ↓ LV ↓ Filesystem ↓ /home A simple /home example with LVM Suppose /home starts with 100 GB:\n/dev/sda → PV ↓ ubuntu-vg ↓ home-lv 100 GB ↓ ext4 ↓ /home Now /home reaches 95 GB used.\nI need another 200 GB.\nInstead of rearranging old partitions, I attach another 200 GB disk:\n/dev/sdb = 200 GB I can give that disk to LVM:\nold PV 100 GB ─┐ ├── ubuntu-vg = 300 GB total capacity new PV 200 GB ─┘ Then the Logical Volume backing /home can grow:\nhome-lv 100 GB → 300 GB The filesystem grows too:\nfilesystem 100 GB → 300 GB And the user still sees:\n/home Nothing about the application path needs to change.\nHow can two different disks look like one storage device? This is the part that can feel strange at first.\nSuppose I have:\n/dev/sda = 100 GB /dev/sdb = 200 GB They are still physically two separate block devices.\nLVM does not magically weld them into one physical disk.\nInstead, LVM creates a mapping layer.\nLVM divides Physical Volumes into chunks called Physical Extents (PEs).\nLogical Volumes are built from corresponding Logical Extents (LEs).\nConceptually:\nPhysical storage: Disk A [ PE1 ][ PE2 ][ PE3 ] Disk B [ PE4 ][ PE5 ][ PE6 ] LVM can map them into one logical sequence:\nLogical Volume: [ LE1 ][ LE2 ][ LE3 ][ LE4 ][ LE5 ][ LE6 ] For example:\nLE1 → PE on Disk A LE2 → PE on Disk A LE3 → PE on Disk A LE4 → PE on Disk B LE5 → PE on Disk B LE6 → PE on Disk B The filesystem above LVM sees one logical block device:\n/dev/ubuntu-vg/home-lv It does not need to care that some blocks ultimately live on one PV and some on another.\nThat is the trick.\nLVM does not physically insert one disk into another LVM is not doing this:\nDisk A [ old data ][ Disk B somehow inserted here ][ more data ] The physical devices remain separate.\nLVM simply maintains a logical map above them:\nDisk A ─┐ ├── VG → LV → filesystem → /home Disk B ─┘ This is why the physical space does not need to be adjacent in the same way a normal partition does.\nLVM does not magically fix a blocked partition There is an important limitation.\nSuppose the PV itself is a partition:\n/dev/sda ├── /dev/sda3 → PV └── /dev/sda4 → another filesystem and free space exists after /dev/sda4.\nLVM cannot make /dev/sda3 jump over /dev/sda4.\nThe normal partition rule still applies.\nThe advantage is that I may not need to grow /dev/sda3 at all.\nI can attach another disk:\n/dev/sdb → new PV and add it to the existing VG:\n/dev/sda3 → old PV ─┐ ├── VG → LV → /home /dev/sdb → new PV ─┘ So LVM does not remove the physical partition limitation.\nIt gives me another way around it.\nA real LVM expansion flow Suppose:\ndf -h /home shows:\nFilesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-home--lv 100G 95G 5G 95% /home I inspect the storage layers:\nlsblk -f pvs vgs lvs Suppose the VG has no free capacity and I attach a new empty 200 GB disk that Linux identifies as /dev/sdb.\nAfter carefully verifying that /dev/sdb is the intended empty disk, the LVM part is conceptually:\npvcreate /dev/sdb vgextend ubuntu-vg /dev/sdb lvextend -l +100%FREE /dev/ubuntu-vg/home-lv Then I grow the filesystem.\nFor ext4:\nresize2fs /dev/ubuntu-vg/home-lv For XFS:\nxfs_growfs /home Afterward:\ndf -h /home may show something like:\nFilesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-home--lv 300G 95G 205G 32% /home The mount point is still exactly the same:\n/home Only the storage layers underneath it changed.\nWhat if I enlarge the existing LVM disk instead? LVM also works when I enlarge an existing virtual disk.\nIf the whole device is already a PV:\n/dev/sdb ↓ PV and Proxmox changes it from:\n100 GB to:\n300 GB then, after Linux sees the larger device, LVM can discover the new capacity with:\npvresize /dev/sdb Then the flow is:\nlarger PV ↓ VG gains free capacity ↓ LV grows ↓ filesystem grows ↓ /home grows If the PV is inside a partition such as /dev/sdb1, that partition may still need to grow first.\nAgain, LVM does not make partition geometry disappear.\nWhy df -h can stay unchanged after I add storage Adding capacity at one layer does not automatically resize every layer above it.\nFor example:\nProxmox disk 300 GB can still have:\npartition 100 GB or:\nPV 100 GB or:\nLV 100 GB or:\nfilesystem 100 GB df -h reports the filesystem’s usable capacity.\nSo I think from bottom to top:\nProxmox / iSCSI ↓ block device ↓ partition, if used ↓ PV, if LVM is used ↓ VG ↓ LV ↓ filesystem ↓ /home Then I ask:\nWhich layer has not grown yet?\nMy usual troubleshooting flow I start with:\ndf -h Then:\nlsblk -f For /home specifically:\nfindmnt /home df -T /home If LVM is involved:\npvs vgs lvs Those commands answer different questions:\ndf -h → which filesystem is running out of space? lsblk -f → what disks, partitions, filesystems, and mounts exist? findmnt /home → what device actually backs /home? pvs → which devices are LVM Physical Volumes? vgs → how much total and free capacity exists in the Volume Group? lvs → how large is the Logical Volume? Non-LVM versus LVM Non-LVM LVM Keep /home as the same path Yes Yes Grow an existing disk Easy when the layout permits Easy when the layout permits Partition needs adjacent free space Yes Yes, if growing that same PV partition Add another disk to the same logical storage Not directly Yes Combine capacity from multiple disks/LUNs Not directly Yes, through PVs and a VG Avoid moving an unrelated blocking partition Often requires migration or rearrangement Often possible by adding another PV Number of storage layers Fewer More Troubleshooting Simpler Requires PV/VG/LV knowledge Simple fixed-purpose server Excellent Sometimes unnecessary Long-lived server that keeps growing Can become layout-dependent Usually more flexible When I am happy with non-LVM Non-LVM is perfectly reasonable when the layout is simple.\nFor example:\n/dev/sda4 ↓ ext4 ↓ /home If /dev/sda4 is the final partition and the virtual disk can be enlarged, growing /home can be very straightforward.\nThat is a good design.\nWhen I prefer LVM I prefer LVM when I expect storage requirements to change over the lifetime of the server.\nA server may start with:\n/home = 100 GB then need:\n/home = 300 GB and later:\n/home = 500 GB The next capacity increase may come from:\na larger Proxmox virtual disk another Proxmox virtual disk an iSCSI LUN another iSCSI LUN LVM gives me a layer where those physical storage changes do not have to map directly to changes in the path the application or user sees.\nLVM is not RAID Combining capacity through LVM does not automatically make the data redundant.\nIf I have:\n/dev/sda ─┐ ├── VG /dev/sdb ─┘ that does not mean every block exists on both devices.\nIt only means both devices can contribute storage capacity.\nIn simple terms:\nLVM → flexible storage management RAID → disk layout and possible redundancy Backup → recovery from data loss They solve different problems.\nThe point The problem I care about is not simply:\nHow do I attach another disk?\nIt is:\n/home is almost full. How do I make the same /home larger with the least disruptive change?\nWithout LVM, that may be very easy when /home is the last partition and the free space is directly after it.\nBut if the layout is:\n[/home][swap][free] or:\n[/home][another filesystem][free] then the partition in the middle has to be dealt with first.\nIf it is swap, it can usually be recreated.\nIf it contains movable data, I can migrate that data first.\nIf it contains important system data, I may prefer migration, maintenance mode, or a different disk instead of live partition surgery.\nLVM exists to reduce how much storage management depends on that physical partition layout.\nIt lets me think in terms of a pool:\nDisk / LUN 1 → PV ─┐ ├── VG → LV → filesystem → /home Disk / LUN 2 → PV ─┘ The underlying devices can change while /home remains /home.\nThat is the practical reason I usually prefer LVM for Linux servers that I expect to keep growing.\n","date":"2026-10-28","description":"A practical explanation of how Linux storage expansion works with and without LVM, why partition layout matters, what happens when another partition blocks /home, and why LVM exists.","id":"when-df-h-is-full-why-i-usually-extend-storage-with-lvm","markdown":"https://ilham.dev/posts/when-df-h-is-full-why-i-usually-extend-storage-with-lvm/index.md","markdown_url":"https://ilham.dev/posts/when-df-h-is-full-why-i-usually-extend-storage-with-lvm/index.md","reading_minutes":15,"summary":"Most storage expansion problems I run into start with something simple:\ndf -h Imagine I see this:\nFilesystem Size Used Avail Use% Mounted on /dev/mapper/ubuntu--vg-root--lv 50G 20G 30G 40% / /dev/sda2 2G 250M 1.","tags":["linux","storage","lvm","proxmox","iscsi","devops"],"title":"When /home Is Full: Why LVM Makes Storage Expansion Easier","url":"https://ilham.dev/posts/when-df-h-is-full-why-i-usually-extend-storage-with-lvm/","word_count":3136},{"canonical_url":"https://ilham.dev/posts/why-the-comments-live-on-github/","content_markdown":"\nThis site is still new, and comments were the last thing I set up. I went with Disqus\nbecause it is the obvious choice: one `\u003cscript\u003e` tag, and replies work without me\nrunning a database or a moderation panel.\n\nThen the post pages started to feel slow. I opened the network panel to see what was\nloading, and there was a lot more on the page than a comment box.\n\n## What was on the page\n\nDisqus is not one script. It is a small ecosystem.\n\n- LiveIntent (`d-code.liadm.com`), an advertising identity script\n- the Facebook SDK (`connect.facebook.net/sdk.js`)\n- Narrative.io, a data marketplace\n- Rezync\n- a Google-hosted copy of Roboto\n- a Google OAuth iframe\n\nBetween them they set six cookies on the page: `disqusauth`, `__jid`, `disqus_unique`,\n`io.narrative.guid.v2`, `zync-uuid`, and `sd-session-id`. Not one of those exists so\nthat a comment can be displayed.\n\nIt also broke the back/forward cache. Lighthouse reported an `unload` handler inside a\nsub-frame — an old advertising pattern that quietly costs you instant back-button\nnavigation.\n\nThe post page weighed about 1,044 KiB. Roughly 6 KiB of that was mine.\n\n## What bothered me\n\nNone of it was for the comments. Every reader who opened a post loaded all of it,\nwhether or not they scrolled down to read a reply.\n\nI do not mind paying for something. I mind not knowing what I am paying with.\n\n## What I moved to\n\nComments now run on [giscus](https://giscus.app). Like Disqus, it is one script tag.\nUnlike Disqus, that is all it is: giscus reads and writes GitHub Discussions through\nthe GitHub API and renders the result in an iframe. There is nothing else in it.\n\nThe comments on a post are a discussion in a public repository. Signing in means\nsigning in with GitHub.\n\n## The trade-offs\n\nI am not going to pretend this costs nothing.\n\n- **Commenting needs a GitHub account.** Some people who would have left a thoughtful\n reply will not, and that is a real loss. For a site like this I think the trade is\n worth it, but it is a trade, not a clear win.\n- **The comments are public and stored elsewhere.** They live in a public repository and\n can be indexed by search engines. If that matters to you, do not use a comment box at\n all — mine included.\n- **It depends on giscus.app, which is one person.** giscus is maintained by a single\n developer, funded by donations, and hosted on Vercel's open-source program. Its own\n README warns that GitHub's Discussions API is still evolving. That is a genuine\n dependency, and I would rather say so than pretend it is not there.\n\nWhat makes it tolerable is that the data is not in giscus. If giscus.app disappeared\ntomorrow, the worst case is that the widget stops rendering. The discussions stay in my\nrepository, readable without it, so nothing anyone wrote would be lost — and if I ever\nwant to drop even that dependency, giscus can be self-hosted.\n\n## Where the page ended up\n\n| | Disqus | giscus |\n| --- | --- | --- |\n| Page weight | ~1,044 KiB | ~124 KiB |\n| Third-party cookies | 6 | 0 |\n| Best Practices | 77 | 100 |\n| Performance | 92 | 99 |\n\nOn the post page, Lighthouse now reports 99 / 100 / 100 / 100, and total blocking time\nfell from about 360 ms to 20 ms. The widget also loads lazily, so the article is\nfinished before the comments even start to arrive.\n\n## The point\n\nDisqus worked. It just did not come alone: an advertising stack, six cookies, and\nmost of a megabyte of JavaScript arrived with it, and none of that was visible from the\noutside.\n\nThe question is rarely whether a tool works. It is what else comes along with it.\nOn a page you control, you can at least look.\n","content_text":"This site is still new, and comments were the last thing I set up. I went with Disqus because it is the obvious choice: one \u003cscript\u003e tag, and replies work without me running a database or a moderation panel.\nThen the post pages started to feel slow. I opened the network panel to see what was loading, and there was a lot more on the page than a comment box.\nWhat was on the page Disqus is not one script. It is a small ecosystem.\nLiveIntent (d-code.liadm.com), an advertising identity script the Facebook SDK (connect.facebook.net/sdk.js) Narrative.io, a data marketplace Rezync a Google-hosted copy of Roboto a Google OAuth iframe Between them they set six cookies on the page: disqusauth, __jid, disqus_unique, io.narrative.guid.v2, zync-uuid, and sd-session-id. Not one of those exists so that a comment can be displayed.\nIt also broke the back/forward cache. Lighthouse reported an unload handler inside a sub-frame — an old advertising pattern that quietly costs you instant back-button navigation.\nThe post page weighed about 1,044 KiB. Roughly 6 KiB of that was mine.\nWhat bothered me None of it was for the comments. Every reader who opened a post loaded all of it, whether or not they scrolled down to read a reply.\nI do not mind paying for something. I mind not knowing what I am paying with.\nWhat I moved to Comments now run on giscus. Like Disqus, it is one script tag. Unlike Disqus, that is all it is: giscus reads and writes GitHub Discussions through the GitHub API and renders the result in an iframe. There is nothing else in it.\nThe comments on a post are a discussion in a public repository. Signing in means signing in with GitHub.\nThe trade-offs I am not going to pretend this costs nothing.\nCommenting needs a GitHub account. Some people who would have left a thoughtful reply will not, and that is a real loss. For a site like this I think the trade is worth it, but it is a trade, not a clear win. The comments are public and stored elsewhere. They live in a public repository and can be indexed by search engines. If that matters to you, do not use a comment box at all — mine included. It depends on giscus.app, which is one person. giscus is maintained by a single developer, funded by donations, and hosted on Vercel’s open-source program. Its own README warns that GitHub’s Discussions API is still evolving. That is a genuine dependency, and I would rather say so than pretend it is not there. What makes it tolerable is that the data is not in giscus. If giscus.app disappeared tomorrow, the worst case is that the widget stops rendering. The discussions stay in my repository, readable without it, so nothing anyone wrote would be lost — and if I ever want to drop even that dependency, giscus can be self-hosted.\nWhere the page ended up Disqus giscus Page weight ~1,044 KiB ~124 KiB Third-party cookies 6 0 Best Practices 77 100 Performance 92 99 On the post page, Lighthouse now reports 99 / 100 / 100 / 100, and total blocking time fell from about 360 ms to 20 ms. The widget also loads lazily, so the article is finished before the comments even start to arrive.\nThe point Disqus worked. It just did not come alone: an advertising stack, six cookies, and most of a megabyte of JavaScript arrived with it, and none of that was visible from the outside.\nThe question is rarely whether a tool works. It is what else comes along with it. On a page you control, you can at least look.\n","date":"2026-09-20","description":"Disqus worked. Then I looked at what else it was loading, and moved the comments to giscus and GitHub Discussions.","id":"why-the-comments-live-on-github","markdown":"https://ilham.dev/posts/why-the-comments-live-on-github/index.md","markdown_url":"https://ilham.dev/posts/why-the-comments-live-on-github/index.md","reading_minutes":3,"summary":"This site is still new, and comments were the last thing I set up. I went with Disqus because it is the obvious choice: one \u0026lt;script\u0026gt; tag, and replies work without me running a database or a moderation panel.","tags":["comments","giscus","performance"],"title":"Why the comments live on GitHub","url":"https://ilham.dev/posts/why-the-comments-live-on-github/","word_count":613},{"canonical_url":"https://ilham.dev/posts/running-services-on-a-small-vps/","content_markdown":"\nA lot of what looks like a scaling problem is really a maintenance problem. On a\nsmall VPS with a gigabyte or two of RAM you can run a surprising amount — as long as\nyou keep the number of moving parts low and make the boring things automatic.\n\n## Keep the request path short\n\nThe fastest request is the one you never have to serve dynamically. Static files,\ncached responses, and a sensible CDN in front of the origin remove most of the load\nbefore it ever reaches your box. What is left is usually small enough to fit.\n\n## Make the machine easy to reason about\n\nA few rules I keep coming back to:\n\n- one job per container, one process per container;\n- explicit resource limits, so one service cannot take the whole box down;\n- health checks and restart policy, so failures are boring;\n- configuration in files under version control, not in a shell history.\n\nResources are finite, so treat them as a budget:\n\n```sh\n# What is actually using memory right now?\ndocker stats --no-stream --format \\\n 'table {{.Name}}\\t{{.MemUsage}}\\t{{.CPUPerc}}'\n```\n\n## Back up the small things that matter\n\nMost of a server is disposable. The database, the configuration, and the certificates\nare not. If you can restore those three, you can rebuild everything else in an\nafternoon.\n\n\u003e Boring, small, and reproducible beats clever, large, and fragile almost every time.\n\nNone of this is exotic. It is just a set of habits that keep a small machine honest —\nand keep you out of the terminal at midnight.\n","content_text":"A lot of what looks like a scaling problem is really a maintenance problem. On a small VPS with a gigabyte or two of RAM you can run a surprising amount — as long as you keep the number of moving parts low and make the boring things automatic.\nKeep the request path short The fastest request is the one you never have to serve dynamically. Static files, cached responses, and a sensible CDN in front of the origin remove most of the load before it ever reaches your box. What is left is usually small enough to fit.\nMake the machine easy to reason about A few rules I keep coming back to:\none job per container, one process per container; explicit resource limits, so one service cannot take the whole box down; health checks and restart policy, so failures are boring; configuration in files under version control, not in a shell history. Resources are finite, so treat them as a budget:\n# What is actually using memory right now? docker stats --no-stream --format \\ 'table {{.Name}}\\t{{.MemUsage}}\\t{{.CPUPerc}}' Back up the small things that matter Most of a server is disposable. The database, the configuration, and the certificates are not. If you can restore those three, you can rebuild everything else in an afternoon.\nBoring, small, and reproducible beats clever, large, and fragile almost every time.\nNone of this is exotic. It is just a set of habits that keep a small machine honest — and keep you out of the terminal at midnight.\n","date":"2026-09-12","description":"You do not need a large machine to run real things. You need a small number of good habits.","id":"running-services-on-a-small-vps","markdown":"https://ilham.dev/posts/running-services-on-a-small-vps/index.md","markdown_url":"https://ilham.dev/posts/running-services-on-a-small-vps/index.md","reading_minutes":2,"summary":"A lot of what looks like a scaling problem is really a maintenance problem. On a small VPS with a gigabyte or two of RAM you can run a surprising amount — as long as you keep the number of moving parts low and make the boring things automatic.","tags":["infrastructure","linux","self-hosting"],"title":"Running real services on a small VPS","url":"https://ilham.dev/posts/running-services-on-a-small-vps/","word_count":254},{"canonical_url":"https://ilham.dev/posts/why-this-site-is-static/","content_markdown":"\nWhen people hear \"blog\" they often picture a database, a login screen, and a\ndashboard full of plugins. You can absolutely run it that way. But if all you are\npublishing is words, a static site is a very good deal.\n\n## The idea is simple\n\nA static site generator reads your content — usually Markdown — and your templates,\nthen writes a folder full of HTML, CSS, and images. That folder is the entire website.\nServing it is just handing files to whoever asks.\n\nThe workflow looks like this:\n\n1. write a Markdown file in `content/posts/`;\n2. run the build;\n3. copy the output to the web root.\n\nThat is the whole pipeline. There is no request-time rendering to reason about.\n\n## What you get\n\n- **Speed**, because there is no application to boot per request.\n- **Security**, because there is no admin panel, no session store, and no database to\n attack.\n- **Portability**, because the output is plain files that any web server can host.\n- **Durability**, because your content is text you own, not rows inside a product.\n\n## What you give up\n\nStatic sites are not the right tool for everything. If you need comments,\npersonalization, or a live editing experience for non-technical writers, you will end\nup bolting those on — or you should pick a different tool on purpose.\n\nFor a personal site, the trade is usually worth it. The content stays in plain text,\nthe deployment stays boring, and the only thing that can really break is the build —\nwhich you run before anything goes live anyway.\n","content_text":"When people hear “blog” they often picture a database, a login screen, and a dashboard full of plugins. You can absolutely run it that way. But if all you are publishing is words, a static site is a very good deal.\nThe idea is simple A static site generator reads your content — usually Markdown — and your templates, then writes a folder full of HTML, CSS, and images. That folder is the entire website. Serving it is just handing files to whoever asks.\nThe workflow looks like this:\nwrite a Markdown file in content/posts/; run the build; copy the output to the web root. That is the whole pipeline. There is no request-time rendering to reason about.\nWhat you get Speed, because there is no application to boot per request. Security, because there is no admin panel, no session store, and no database to attack. Portability, because the output is plain files that any web server can host. Durability, because your content is text you own, not rows inside a product. What you give up Static sites are not the right tool for everything. If you need comments, personalization, or a live editing experience for non-technical writers, you will end up bolting those on — or you should pick a different tool on purpose.\nFor a personal site, the trade is usually worth it. The content stays in plain text, the deployment stays boring, and the only thing that can really break is the build — which you run before anything goes live anyway.\n","date":"2026-08-30","description":"Markdown in, HTML out — and why that is still a good deal in 2026.","id":"why-this-site-is-static","markdown":"https://ilham.dev/posts/why-this-site-is-static/index.md","markdown_url":"https://ilham.dev/posts/why-this-site-is-static/index.md","reading_minutes":2,"summary":"When people hear \u0026ldquo;blog\u0026rdquo; they often picture a database, a login screen, and a dashboard full of plugins. You can absolutely run it that way. But if all you are publishing is words, a static site is a very good deal.","tags":["hugo","static-sites"],"title":"Why this site is a static site","url":"https://ilham.dev/posts/why-this-site-is-static/","word_count":255}],"schema_version":"1.0","url":"https://ilham.dev/posts/"}