xfs filesystem full

11 February 2012 - Fairbanks

Ran into an interesting problem with a large XFS volume today where it ran out of space when it actually had plenty of free space. Turns out this is an odd issue related unable to allocate any further inodes.

xfs dynamically allocates inodes though!

Yes, but apparently it dynamically allocates inodes and keeps track of that allocation in the first 1TB of space. If you have completely filled that first 1TB it can’t dynamically allocation more causing a ‘No space left on device’ error.

[dbroders@soy SDMI.ORTHO.2010]$ df -h .
Filesystem            Size  Used Avail Use% Mounted on
                      2.7T  2.4T  401G  86% /data/scratch
[dbroders@soy SDMI.ORTHO.2010]$ touch asdf
touch: cannot touch `asdf': No space left on device

Enter the google where I found a wonderful run down on how the problem and how to solve it:

[root@soy ~]# xfs_info /data/scratch/
meta-data=/dev/mapper/md0raid0-work isize=256    agcount=87, agsize=8388480 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=724697088, imaxpct=5
         =                       sunit=128    swidth=256 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

agsize 8388480 and bsize is 4K … yadda yadda what is the solution.

I picked a particular directory I knew held data from a long time ago to simplify my search, my goal to find files in Allocation Group 0. So I made the following silly ruby script:

#!/usr/bin/env ruby
target_dir = ARGV.shift
require 'find'
Find.find(target_dir) do |p|
  next unless File.file?(p)
  r = `xfs_bmap -v '#{p}'`
  ag = r.split("\n")[2].split[3]
  puts p if ag == "0"

I then just grabbed some of the larger files in that list. Moved (mv) them to /tmp and then moved them back again. Life was good.