A dedicated server's filesystem reporting full is rarely as straightforward as it appears. The instinctive response — deleting large files or expanding the volume — often misses the actual cause entirely. Disk space alerts can fire even when gigabytes of data have already been removed, and the standard tools most engineers reach for first will not always tell the whole story.
Three causes account for the majority of unexpected filesystem exhaustion on dedicated servers, and all three are routinely misdiagnosed. Inode depletion can make a filesystem appear full even when block space remains available. Runaway log accumulation can consume hundreds of gigabytes before any alert threshold is crossed.
Deleted files held open by a running process can silently retain allocated space that neither the filesystem nor the operating system will release until the process is restarted. Each of these scenarios produces symptoms that look identical on the surface but require a completely different remediation path.
This article walks through the diagnostic logic behind each cause — what the filesystem is actually reporting, why the standard output of common disk-inspection commands can be misleading, and which follow-up commands surface the real offender.
What Is Dedicated Server Disk Space and Why It Fills Up Unexpectedly
A dedicated server filesystem tracks consumed space across two independent accounting systems: block allocation and inode allocation. Blocks store the actual data — file contents, database records, log entries. Inodes store the metadata that describes each file: its name, permissions, ownership, and the pointer to its blocks. When either resource is exhausted, the operating system reports the filesystem as full, even if the other resource still has capacity to spare.
This dual-accounting model is the source of most diagnostic confusion. An engineer running a standard disk-usage command sees available block space and concludes the alert must be a false positive. But if the inode table is saturated — which happens when a directory contains millions of tiny files, such as cache entries or mail queue items — no new file can be created regardless of how many gigabytes of block space remain unused.
The operating system cannot allocate a new inode, so every write attempt fails with a "no space left on device" error. The block-level view looks healthy; the inode-level view tells a different story entirely.
Filesystem accounting also behaves differently depending on how the partition was formatted. The inode count is fixed at format time on most traditional Linux filesystems. Once that ceiling is reached, it cannot be raised without reformatting the partition — which is why catching inode exhaustion early matters far more than it might appear.
Modern copy-on-write filesystems handle inodes differently, but the majority of production dedicated servers still run on ext4 or xfs, where this constraint applies directly.
A practical complication arises with reserved block space. Many filesystems reserve a percentage of total blocks for the root user, meaning non-root processes will encounter "disk full" errors while the filesystem technically still holds free capacity. Understanding which layer is failing — blocks, inodes, or reserved headroom — determines every subsequent diagnostic step.
A process clinging to a deleted file descriptor can silently consume gigabytes that standard directory scans will never reveal.
Why df and du Contradict Each Other – Understanding the Diagnostic Gap
When df and du report different numbers, the most common cause is a deleted-but-open file handle: a process still holds a file descriptor to a file that has already been removed from the directory tree. The operating system cannot release the disk blocks that file occupies until every open file descriptor pointing to it is closed. From du's perspective, the file is gone — it no longer appears in any directory, so du skips it entirely.
From df's perspective, those blocks are still in use, because the kernel's block accounting follows file descriptors, not directory entries.
This distinction matters immediately in practice. An engineer deletes a multi-gigabyte log file, confirms it no longer appears in any directory listing, and expects df to reflect the recovered space. It does not. The application that originally opened that file — a web server, a database process, or a monitoring agent — is still writing to it through its cached file descriptor.
The blocks remain allocated until that process is restarted or the descriptor is explicitly closed. The filesystem reports no change, the engineer suspects a bug, and the real cause goes unaddressed for hours.
A second structural gap exists at the kernel accounting level. The df command queries the filesystem's own superblock statistics, which include reserved blocks and account for all open descriptors. The du command walks directory trees and sums the apparent sizes of reachable files. Neither tool is wrong; they are simply measuring different things.
Reserved block headroom — which the previous section established as a distinct layer — contributes to this gap as well, since df subtracts reserved space from the available figure while du ignores it entirely.
Recognizing which gap applies to your situation determines the correct next step. Chasing directory sizes with du when the real culprit is a deleted-but-open descriptor wastes diagnostic time and risks unnecessary service disruption.
How Inode Exhaustion Fills a Disk Without Consuming Block Space
Inode exhaustion produces a “no space left on device” error even when block usage is low. Every file on a Linux filesystem — regardless of its size — consumes exactly one inode, a metadata record that stores ownership, permissions, and block pointers. The total number of inodes is fixed at filesystem creation time. Once that pool is depleted, the system cannot create new files, even if gigabytes of block space remain completely untouched.
A mail queue backlog of hundreds of thousands of messages can drain your inode pool while barely touching block storage.
Certain workload patterns burn through inodes at a rate that surprises even experienced engineers. Mail transfer agents that spool each message as a separate file are a classic trigger: a queue backlog of several hundred thousand messages consumes an equivalent number of inodes while occupying relatively modest block space.
PHP session stores that write one file per active session, thumbnail caches that generate multiple derivative images per upload, and package managers that extract archives into directories containing thousands of small files all share the same pattern. The common thread is high-frequency small-file creation with infrequent cleanup. A single overnight job that generates temporary files without removing them can silently exhaust the inode pool before the morning shift arrives.
The diagnostic command that surfaces this is straightforward: running df with the inode flag displays inode totals, used counts, and available counts for every mounted filesystem. A filesystem showing ninety-nine percent inode utilization alongside thirty percent block utilization is a clear signal that block-focused cleanup will accomplish nothing.
The correct response is to identify which directory holds the largest number of files — not the largest total size — and that requires a recursive file-count query rather than a standard disk-usage scan. These two tools measure fundamentally different dimensions, and confusing file count with file size is the diagnostic mistake that delays resolution longest.
The danger of runaway log growth lies not in how large a directory has become, but in how fast a single file inside it is still expanding.
How to Identify Runaway Log Accumulation as the Real Space Consumer
Runaway log accumulation stands apart from other disk exhaustion causes because its diagnostic signature is rate-based rather than size-based. A large log directory is not automatically a problem; the warning sign is a single, continuously expanding file with no compressed archives beside it and no rotation timestamp in recent history. That pattern confirms rotation has failed silently, not merely that the application is verbose.
The practical threshold worth acting on is any individual log file that has grown beyond the expected maximum for a single rotation interval — commonly several gigabytes for a busy web server access log. When that is also the only file in its directory, with no accompanying compressed archives, the rotation job has almost certainly crashed and left no alert behind. That silent failure mode is what makes log bloat the hardest of the three space consumers to catch before it becomes critical.
The first directory to inspect is the system log path, typically located under the var directory on Linux systems. Within that path, look specifically for any single file exceeding several gigabytes. A web server access log that has grown to forty or fifty gigabytes is a reliable indicator that log rotation has either been disabled, is misconfigured, or has silently failed.
Crashed rotation jobs are a common culprit: the rotation process exits with an error, leaves the active log file untouched, and the failure is never surfaced to an alert channel. Application-level debug modes cause a parallel problem. When a developer enables verbose output to chase a transient bug and forgets to revert the setting, the logging volume can increase by an order of magnitude overnight. Debug-mode log floods of this kind can fill a filesystem within hours on a high-traffic server.
The second area to inspect is application-specific log paths outside the system directory — database query logs, mail transfer agent logs, and custom application log directories that rotation policies sometimes miss entirely. Listing files by modification time rather than alphabetically reveals which files are still actively growing. Cross-directory log audits conducted on a scheduled basis catch these outliers before they become emergencies.
Once a file is unlinked from the directory tree but held open by a running process, the space it occupies becomes invisible to every tool that navigates folders rather than file descriptors.
How Deleted-but-Open File Handles Hide Gigabytes from du
This means du, which traverses the directory tree, reports the file as gone. The blocks, however, remain allocated. The result is a gap between what du counts and what df reports as used: gigabytes of space consumed by files that no longer appear anywhere in the filesystem.
The practical consequence is significant. A log rotation job may delete a multi-gigabyte log file, but if the application that writes to it is still running and holds an open descriptor, the space is not returned to the filesystem. The same pattern appears with database engines that delete their own temporary files during startup, and with backup agents that remove staging files while a transfer is still in progress.
In each case, the process table retains the handle, the kernel retains the blocks, and the operator sees a filesystem that refuses to free space despite apparently having nothing large left to delete.
The tool that surfaces these phantom file handles is the open-file listing utility available on Linux systems, which can filter output to show only deleted files still held open by a running process. Each result includes the process identifier, the file descriptor number, and the size of the space still allocated.
For files that cannot wait — where the holding process is a long-running daemon — redirecting the file descriptor to a null device via the process filesystem path releases the blocks without terminating the process and without a reboot.
Which Hidden Space Consumers Are Most Frequently Overlooked
The space consumers addressed here share a different characteristic: they survive standard directory scans entirely, meaning neither du nor a visual inspection of top-level directories will surface them without deliberate effort.
A large gap between df and du output is your clearest signal that bind mounts or package caches are hiding the missing space.
The deciding factor for prioritising which of these to investigate first is whether the discrepancy between df and du output is large or small. A significant gap — where df reports far less free space than du totals would predict — points toward bind mounts shadowing pre-mount files or package manager caches stored outside the scanned tree. A negligible gap alongside an unexpectedly high file count instead suggests temporary directories accumulating small files that persist across reboots.
- Bind mounts that shadow existing directories, making pre-mount files invisible to du while df still counts their blocks
- Temporary file directories that survive reboots and accumulate unmanaged data over time
- Package manager caches that grow silently across months of routine system updates
- Files deleted by log rotation jobs but still held open by running application processes
- Database engines that delete their own temporary files at startup yet retain open descriptors until restart
- Backup agents that remove staging files from the directory tree without releasing block allocations
A bind mount maps one directory path onto another location in the filesystem tree. When a directory is mounted over an existing path, any files written beneath that path before the mount occurred are still present on the underlying device — they simply become invisible to directory traversal tools. The result is that blocks are consumed and counted by df, but no scan of the visible filesystem will surface them.
Unmounting the path temporarily and re-running a directory size audit is the only reliable way to expose what lies beneath. This scenario appears most often on servers where container runtimes or chroot environments have been configured and later partially removed, leaving orphaned mount points behind.
Temporary file directories present a different problem. On many Linux configurations, the system temporary directory is memory-backed and does not survive a reboot — but a secondary temporary path used by specific services may be disk-backed and subject to no automatic cleanup. Application frameworks that write session data, upload staging files, or intermediate build artifacts to disk-backed temporary paths can fill those directories over weeks without triggering any alert.
Listing directory sizes across all disk-backed temporary paths, not just the standard system location, is a necessary step in any thorough sweep.
Package manager caches deserve equal attention. Every installed or upgraded package leaves compressed archive files in a local cache directory. On a server that has been in production for a year or more, this cache can occupy several gigabytes. The cache serves no operational function once packages are installed, yet it is rarely included in automated cleanup routines.
A one-time fix that leaves monitoring and log rotation unchanged is simply a countdown to the next identical crisis.
How to Prevent Filesystem Exhaustion from Recurring After You Resolve It
Resolving a filesystem crisis once is not enough. Without structural changes to monitoring and log management, the same exhaustion pattern will return — often faster than the first time, because the underlying process that caused it is still running unchanged.
- Set independent threshold alerts for block usage and inode usage — a single combined alert is blind to inode exhaustion
- Alert at 75% utilization as the investigation trigger and at 90% as the mandatory action threshold
- Configure logrotate with explicit size caps and compression to prevent any single log file from growing unbounded
- Automate periodic scans for deleted-but-open file handles using lsof or equivalent tooling
- Schedule regular audits of package manager cache directories and enforce size limits or automatic pruning
- Monitor inode consumption trends over time, not just point-in-time snapshots, to catch workloads that generate millions of tiny files
- Overview and test log rotation configurations after any application deployment or configuration change
The most durable prevention starts with threshold-based alerting on both block usage and inode usage independently. A common operational mistake is alerting only on block consumption. Because inode exhaustion can occur while blocks remain plentiful, a single disk-space alert is structurally blind to half the failure modes covered in this article. Setting separate alerts at 75% and 90% for each accounting system gives the team two intervention windows before a service-impacting event.
The 75% threshold is the signal to investigate; the 90% The signal to act immediately.
Log rotation policy is the second structural change that prevents recurrence. Reactive cleanup — manually deleting oversized log files after an incident — addresses the symptom without touching the cause. A durable policy defines maximum file size, maximum retention period, and a compression schedule for every log-producing service on the server, not just the system logger.
Services added later in the server's lifecycle should be enrolled in the same rotation framework at deployment time, not retroactively after they cause a problem.
Disk-backed temporary directories and package manager caches benefit from scheduled automated cleanup on a weekly cadence. Adding them explicitly to a maintenance schedule closes the gap that manual sweeps leave open.
Conclusion – Diagnose the Real Cause Before You Delete Anything
Filesystem exhaustion on a dedicated server almost always has a precise, identifiable cause — but the standard tools surface only part of the picture. When block counts and directory sizes appear to contradict each other, the real culprit is typically inode depletion, a runaway log accumulation, or a deleted file held open by a live process.
Working through each layer in sequence — starting with inode accounting, then log growth, then open file handles, and finally shadowed mount points and silent cache accumulation — eliminates guesswork and prevents the destructive mistake of deleting the wrong data under pressure. Systematic isolation before any deletion is the discipline that separates a controlled recovery from an extended incident.
The diagnostic sequence and alerting thresholds in this article form a repeatable operational foundation. Prevent recurring log-driven exhaustion by implementing the retention and rotation controls in Dedicated Server Log Management — rsyslog and logrotate Setup.
