n1docker存储空间不足

复现

root@aml:~# docker rmi 4b1395db74d8
Error response from daemon: write /var/lib/docker/image/overlay2/.tmp-repositories.json960701718: no space left on device
root@aml:~# docker restart ttnode
Error response from daemon: Cannot restart container ttnode: failed to create OCI runtime console socket: mkdir /tmp/pty422905979: no space left on device: unknown

查看存储情况

• overlay use 100%

root@aml:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            794M     0  794M   0% /dev
tmpfs           185M   24M  162M  13% /run
/dev/mmcblk1p2  6.4G  6.4G     0 100% /
tmpfs           924M     0  924M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           924M     0  924M   0% /sys/fs/cgroup
/dev/sda1        58G   51G  6.3G  90% /mnt/test
armbian-ramlog   50M   39M   12M  77% /var/log
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/eda3df9f161b1a136db88cb55f1bed6c9a542b450ed09f1d95c69f844e512c8f/merged
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/6ed8e78712ae0e54d47643a981b29c3a1a0657ddcac26111c75d84d78063df18/merged
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/1eb97ffd2550970e3cc7ba791fc9d7dfed875b4000ee12d368286a710a079727/merged
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/99b674863422d16171b03702d7752149ceb711e8e2a0f5f736319c6ac8661c26/merged
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/e9986f3c91b768635909613fb8206cf4e9fbc3a7e3a1733ebf1e06ce6521941b/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ff96f1aaeb31ce2c5a70458812c26f0a07c602b38b3f92e2603e0600a9ae94ad/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/c4cc904b5007ea33ac98f18c4df9136af3049281a9287568a66640ef0d6d4294/mounts/shm
overlay         6.4G  6.4G     0 100% /var/lib/docker/overlay2/3ce2a690a941d4bf6f5fb4542303fa24530d98e02295e56cf01a852d98620d6b/merged
tmpfs           185M     0  185M   0% /run/user/0

root@aml:~# docker volume ls
DRIVER              VOLUME NAME
local               5b1b2e1098fdb7b2dc6b66dea35820adab9fd2c91b4ca26c0526f12d4526b79b
local               45c99f47b64623786f5ad9db70e70aa3b51b3da935989322bbe59cc48ddb5bd3
local               94a4c6db4b2721b322acba328a00948dac4f1df9ab78754b1c8e94395c6d0176
local               4529d07658046b61e0000eacdc680da3497fa5c5599e36172f6184b2349a8949
local               ab760444488ee52ea9c46be2813c30fbc84539147450f20a5fb7d7bd414d11d9
local               df7a0c4bdba522ba67db42fa8bc0a3a75eaa87f921eaa68f2eefb01d3e8c0fd7
local               e9baa1b6914156a13901a4d2056c7aed3e50282df68663a6d0a67c6a8ca2f124
local               portainer_data

root@aml:~# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              15                  8                   1.762GB             810.4MB (46%)
Containers          8                   8                   9.227MB             0B (0%)
Local Volumes       8                   1                   577.3MB             577.1MB (99%)
Build Cache         0                   0                   0B                  0B

查看docker 存储空间

root@aml:~# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              16                  9                   2.14GB              771MB (36%)
Containers          9                   8                   58.62MB             0B (0%)
Local Volumes       8                   1                   577.3MB             577.1MB (99%)
Build Cache         0                   0                   0B                  0B

先删除一些镜像

• 如果删除镜像都报错，那就随便找一个10m以上的文件删除，不然命令都执行不了
• 删除一些空间后发现overlay Avail 有279M可以用，暂时解决燃眉之急

root@aml:~# docker rmi 646928661a48
Deleted: sha256:646928661a480dd75c75651e3905beffa1b084d4cab9c4ff99b641be85c80110
Deleted: sha256:5746205a39886b3d7d4c25daf287cdb541c0f5a993201a776b63329a0d4d6593
Deleted: sha256:b8d1c29c8992e7c62d584818a9e6dafeccced31a09f412e7fb7e38dae372df64
Deleted: sha256:b159a894d5b7aef0aa66458f9f2a9ae37763998d813ea0229c5d5d3d9f64904e

root@aml:/opt/root# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            794M     0  794M   0% /dev
tmpfs           185M   24M  162M  13% /run
/dev/mmcblk1p2  6.4G  6.1G  279M  96% /
tmpfs           924M     0  924M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           924M     0  924M   0% /sys/fs/cgroup
/dev/sda1        58G   51G  6.3G  90% /mnt/test
armbian-ramlog   50M  4.7M   46M  10% /var/log
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/eda3df9f161b1a136db88cb55f1bed6c9a542b450ed09f1d95c69f844e512c8f/merged
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/6ed8e78712ae0e54d47643a981b29c3a1a0657ddcac26111c75d84d78063df18/merged
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/1eb97ffd2550970e3cc7ba791fc9d7dfed875b4000ee12d368286a710a079727/merged
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/99b674863422d16171b03702d7752149ceb711e8e2a0f5f736319c6ac8661c26/merged
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/e9986f3c91b768635909613fb8206cf4e9fbc3a7e3a1733ebf1e06ce6521941b/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ff96f1aaeb31ce2c5a70458812c26f0a07c602b38b3f92e2603e0600a9ae94ad/mounts/shm
shm              64M     0   64M   0% /var/lib/docker/containers/c4cc904b5007ea33ac98f18c4df9136af3049281a9287568a66640ef0d6d4294/mounts/shm
overlay         6.4G  6.1G  279M  96% /var/lib/docker/overlay2/3ce2a690a941d4bf6f5fb4542303fa24530d98e02295e56cf01a852d98620d6b/merged
tmpfs           185M     0  185M   0% /run/user/0

迁移docker 存储数据到u盘

• 首先得挂在一个u盘,u盘格式建议exFat
• 下面u盘的位置在Disk /dev/sda

root@aml:/opt/root# fdisk -l
Disk /dev/mmcblk1: 7.3 GiB, 7818182656 bytes, 15269888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc2a59100

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk1p1      1368064  1617919   249856  122M  c W95 FAT32 (LBA)
/dev/mmcblk1p2      1619968 15269887 13649920  6.5G 83 Linux


Disk /dev/sda: 57.3 GiB, 61530439680 bytes, 120176640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1          32 120176639 120176608 57.3G  c W95 FAT32 (LBA)

• 挂载u盘到目录/mnt
  • 这里的-v vfat看个人u盘什么格式类型，我这里的是exfat

mount -t vfat /dev/sda1 /mnt/upan

• 设置开机自动挂载

root@aml:/opt/root# cat /etc/fstab
/dev/sda1 /mnt/test vfat defaults 0 0

• 迁移docker到u盘中
• 查看docker所在目录在哪

root@aml:/opt/root#  docker info | grep "Docker Root Dir"
 Docker Root Dir: /var/lib/docker

root@aml:/opt/root# mkdir -p /mnt/upan/docker

root@aml:/opt/root# cp -a /var/lib/docker /mnt/upan/docker
# 或者使用rsync
root@aml:/opt/root# rsync -avzP /var/lib/docker  /mnt/upan/docker

# 保存一份备份，没有空间就不要搞了
mv /var/lib/docker/ /var/lib/docker.bak

修改docker启动目录为挂载目录

• 方法一：docker版本17.2以上的支持修改/etc/docker/daemon.json文件
  • 加上配置”data-root”: “/mnt/upan/docker”,就行其他的不要动

root@aml:/opt/root# cat /etc/docker/daemon.json
{
  "data-root": "/mnt/upan/docker",
  "debug":true,
  "insecure-registries": ["reg.wsppx.cn"],
  "experimental": true,
  "log-driver":"json-file",
  "log-opts": {"max-size":"900m", "max-file":"3"}
}

• 方法二:软链接到/var/lib/docker

systemctl stop docker

ln -fs /mnt/upan/docker /var/lib/docker

systemctl start docker

• 方法三:修改启动文件
  • 修改这句：ExecStart=/usr/bin/dockerd –graph=/mnt/upan/docker -H fd:// –containerd=/run/containerd/containerd.sock

root@aml:/opt/root# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-10-31 12:22:20 CST; 5 days ago
     Docs: https://docs.docker.com
 Main PID: 2306 (dockerd)
    Tasks: 62
   Memory: 109.7M
      CPU: 6h 54min 51.672s

root@aml:/opt/root# cat /lib/systemd/system/docker.service
[Service]
Type=notify
ExecStart=/usr/bin/dockerd --graph=/mnt/upan/docker -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

如果出现错误

错误一；

在挂载目录创建文件夹报错，没有写入权限，可以执行下面操作

sudo mount -o remount,rw /dev/sda1 /mnt/upan

• 错误二
  • fat格式不支持这个

cp: cannot create symbolic link '/mnt/test/docker/docker/overlay2/647fbbdf639d05aa9910e3c3567f086934cf53e2e99feee1a3fbf0c83c8e2813/diff/usr/lib/libzen.so.0': Operation not permitted

你把驱动器格式化为FAT了吗？那这就是你的问题。当Veracrypt要求您时，将其格式化为NTFS而不是FAT，然后重试。

FAT32 unfortunately does not support owner, permissions or symlinks. That is why you are getting an Operation not permitted error. Note that this error is not “permission denied”: “operation not permitted” often means “operation not supported”, i.e. nobody can do this.