xfs filesystem full
11 February 2012 - Fairbanks
Ran into an interesting problem with a large XFS volume today where it ran out of space when it actually had plenty of free space. Turns out this is an odd issue related unable to allocate any further inodes.
xfs dynamically allocates inodes though!
Yes, but apparently it dynamically allocates inodes and keeps track of that allocation in the first 1TB of space. If you have completely filled that first 1TB it can’t dynamically allocation more causing a ‘No space left on device’ error.
[dbroders@soy SDMI.ORTHO.2010]$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/md0raid0-work
2.7T 2.4T 401G 86% /data/scratch
[dbroders@soy SDMI.ORTHO.2010]$ touch asdf
touch: cannot touch `asdf': No space left on device
Enter the google where I found a wonderful run down on how the problem and how to solve it: http://osvault.blogspot.com/2011/03/fixing-1tbyte-inode-problem-in-xfs-file.html
[root@soy ~]# xfs_info /data/scratch/
meta-data=/dev/mapper/md0raid0-work isize=256 agcount=87, agsize=8388480 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=724697088, imaxpct=5
= sunit=128 swidth=256 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=131072, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0agsize 8388480 and bsize is 4K … yadda yadda what is the solution.
I picked a particular directory I knew held data from a long time ago to simplify my search, my goal to find files in Allocation Group 0. So I made the following silly ruby script:
#!/usr/bin/env ruby
target_dir = ARGV.shift
require 'find'
Find.find(target_dir) do |p|
next unless File.file?(p)
r = `xfs_bmap -v '#{p}'`
ag = r.split("\n")[2].split[3]
puts p if ag == "0"
endI then just grabbed some of the larger files in that list. Moved (mv) them to /tmp and then moved them back again. Life was good.