* [PATCH] nilfs-utils: Work around uncleanable full filesystem
@ 2013-01-14 15:54 Sven Eckelmann
[not found] ` <1358178899-26347-1-git-send-email-sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
0 siblings, 1 reply; 14+ messages in thread
From: Sven Eckelmann @ 2013-01-14 15:54 UTC (permalink / raw)
To: linux-nilfs-u79uwXL29TY76Z2rM5mHXA; +Cc: Sven Eckelmann
The filesystem can end up in a state were the filesystem is full and the
returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
segments. The garbage collector will not clean anything and therefore no new
room for new files will be available and ss_nongc_ctime/sui_lastmod will not be
updated without using special tools. This makes the filesystem unusable without
manual recovery.
Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
--
This problem appeared on a current 3.2 stable kernel (Debian Wheezy build). I
am not an FS developer and have therefore not much background knowledge about
the NILFS codebase. Nevertheless, this problem hit me quite hard after creating
some files on a nilfs partition until it was full and deleting them again.
$ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
$ rm foo*
Looking at the output debugging output using
$ watch -n .5 'df -h;tail /var/log/syslog;'
clearly showed that it was not finding any segments to delete. The only problem
I could find was the threshold. After "removing" this threshold, I was able to
get some clear segments again. I personally cannot explain why the check is
there at all. Maybe there is a good reason but the comment above it didn't help
much.
So, here for completeness the threshold: 1358164666 (aka: Mon Jan 14 12:57:46
CET 2013)
And here are the output of lssu and lscp:
$ lssu --all
SEGNUM DATE TIME STAT NBLOCKS
0 2013-01-14 12:58:23 -d- 2047
1 2013-01-14 12:58:23 -d- 2048
2 2013-01-14 12:58:23 -d- 2048
3 2013-01-14 12:58:26 -d- 2048
4 2013-01-14 12:58:26 -d- 2048
5 2013-01-14 12:58:26 -d- 2048
6 2013-01-14 12:58:26 -d- 2048
7 2013-01-14 12:58:26 -d- 2048
8 2013-01-14 12:58:26 -d- 2048
9 2013-01-14 12:58:26 -d- 2048
10 2013-01-14 12:58:26 -d- 2048
11 2013-01-14 12:58:29 -d- 2048
12 2013-01-14 12:58:29 -d- 2048
13 2013-01-14 12:58:29 -d- 2048
14 2013-01-14 12:58:29 -d- 2048
15 2013-01-14 12:58:29 -d- 2048
16 2013-01-14 12:58:29 -d- 2048
17 2013-01-14 12:58:29 -d- 2048
18 2013-01-14 12:58:29 -d- 2048
19 2013-01-14 12:58:32 -d- 2048
20 2013-01-14 12:58:32 -d- 2048
21 2013-01-14 12:58:32 -d- 2048
22 2013-01-14 12:58:33 -d- 2048
23 2013-01-14 12:58:33 -d- 2048
24 2013-01-14 12:58:33 -d- 2048
25 2013-01-14 12:58:33 -d- 2048
26 2013-01-14 12:58:33 -d- 2048
27 2013-01-14 12:58:36 -d- 2048
28 2013-01-14 12:58:36 -d- 2048
29 2013-01-14 12:58:36 -d- 2048
30 2013-01-14 12:58:36 -d- 2048
31 2013-01-14 12:58:36 -d- 2048
32 2013-01-14 12:58:36 -d- 2048
33 2013-01-14 12:58:36 -d- 2048
34 2013-01-14 12:58:36 -d- 2048
35 2013-01-14 12:58:40 -d- 2048
36 2013-01-14 12:58:40 -d- 2048
37 2013-01-14 12:58:40 -d- 2048
38 2013-01-14 12:58:40 -d- 2048
39 2013-01-14 12:58:40 -d- 2048
40 2013-01-14 12:58:40 -d- 2048
41 2013-01-14 12:58:40 -d- 2048
42 2013-01-14 12:58:40 -d- 2048
43 2013-01-14 12:58:43 -d- 2048
44 2013-01-14 12:58:43 -d- 2048
45 2013-01-14 12:58:43 -d- 2048
46 2013-01-14 12:58:43 -d- 2048
47 2013-01-14 12:58:43 -d- 2048
48 2013-01-14 12:58:43 -d- 2048
49 2013-01-14 12:58:43 -d- 2048
50 2013-01-14 12:58:43 -d- 2048
51 2013-01-14 12:58:47 -d- 2048
52 2013-01-14 12:58:47 -d- 2048
53 2013-01-14 12:58:47 -d- 2048
54 2013-01-14 12:58:47 -d- 2048
55 2013-01-14 12:58:47 -d- 2048
56 2013-01-14 12:58:47 -d- 2048
57 2013-01-14 12:58:47 -d- 2048
58 2013-01-14 12:58:47 -d- 2048
59 2013-01-14 12:58:50 -d- 2048
60 2013-01-14 12:58:51 -d- 2048
61 2013-01-14 12:58:51 -d- 2048
62 2013-01-14 12:58:51 -d- 2048
63 2013-01-14 12:58:51 -d- 2048
64 2013-01-14 12:58:51 -d- 2048
65 2013-01-14 12:58:51 -d- 2048
66 2013-01-14 12:58:51 -d- 2048
67 2013-01-14 12:58:54 -d- 2048
68 2013-01-14 12:58:54 -d- 2048
69 2013-01-14 12:58:54 -d- 2048
70 2013-01-14 12:58:54 -d- 2048
71 2013-01-14 12:58:54 -d- 2048
72 2013-01-14 12:58:54 -d- 2048
73 2013-01-14 12:58:54 -d- 2048
74 2013-01-14 12:58:54 -d- 2048
75 2013-01-14 12:58:57 -d- 2047
76 2013-01-14 12:58:57 -d- 2048
77 2013-01-14 12:58:57 -d- 2048
78 2013-01-14 12:58:57 -d- 2048
79 2013-01-14 12:58:57 -d- 2048
80 2013-01-14 12:58:57 -d- 2048
81 2013-01-14 12:58:57 -d- 2048
82 2013-01-14 12:58:57 -d- 2048
83 2013-01-14 12:59:01 -d- 2048
84 2013-01-14 12:59:01 -d- 2048
85 2013-01-14 12:59:01 -d- 2048
86 2013-01-14 12:59:01 -d- 2048
87 2013-01-14 12:59:01 -d- 2048
88 2013-01-14 12:59:01 -d- 2048
89 2013-01-14 12:59:01 -d- 2048
90 2013-01-14 12:59:01 -d- 2048
91 2013-01-14 12:59:04 -d- 2048
92 2013-01-14 12:59:04 -d- 2048
93 2013-01-14 12:59:04 -d- 2048
94 2013-01-14 12:59:04 -d- 2048
95 2013-01-14 12:59:04 -d- 2048
96 2013-01-14 12:59:04 -d- 2048
97 2013-01-14 12:59:04 -d- 2048
98 2013-01-14 12:59:04 -d- 2048
99 2013-01-14 12:59:08 -d- 2048
100 2013-01-14 12:59:08 -d- 2048
101 2013-01-14 12:59:08 -d- 2048
102 2013-01-14 12:59:08 -d- 2048
103 2013-01-14 12:59:08 -d- 2048
104 2013-01-14 12:59:08 -d- 2048
105 2013-01-14 12:59:08 -d- 2048
106 2013-01-14 12:59:08 -d- 2048
107 2013-01-14 12:59:11 -d- 2048
108 2013-01-14 12:59:11 -d- 2048
109 2013-01-14 12:59:11 -d- 2048
110 2013-01-14 12:59:11 -d- 2048
111 2013-01-14 12:59:11 -d- 2048
112 2013-01-14 12:59:12 -d- 2048
113 2013-01-14 12:59:12 -d- 2048
114 2013-01-14 12:59:12 -d- 2048
115 2013-01-14 12:59:15 -d- 2048
116 2013-01-14 12:59:15 -d- 2048
117 2013-01-14 12:59:15 -d- 2048
118 2013-01-14 12:59:15 -d- 2048
119 2013-01-14 12:59:15 -d- 2048
120 2013-01-14 12:59:15 -d- 2048
121 2013-01-14 12:59:15 -d- 2048
122 2013-01-14 12:59:15 -d- 2048
123 2013-01-14 12:59:18 -d- 2048
124 2013-01-14 12:59:18 -d- 2048
125 2013-01-14 12:59:18 -d- 2048
126 2013-01-14 12:59:18 -d- 2048
127 2013-01-14 12:59:18 -d- 2048
128 2013-01-14 12:59:18 -d- 2048
129 2013-01-14 12:59:18 -d- 2048
130 2013-01-14 12:59:18 -d- 2048
131 2013-01-14 12:59:21 -d- 2048
132 2013-01-14 12:59:21 -d- 2048
133 2013-01-14 12:59:21 -d- 2048
134 2013-01-14 12:59:21 -d- 2048
135 2013-01-14 12:59:21 -d- 2048
136 2013-01-14 12:59:21 -d- 2048
137 2013-01-14 12:59:22 -d- 2048
138 2013-01-14 12:59:22 -d- 2048
139 2013-01-14 12:59:25 -d- 2048
140 2013-01-14 12:59:25 -d- 2048
141 2013-01-14 12:59:25 -d- 2048
142 2013-01-14 12:59:25 -d- 2048
143 2013-01-14 12:59:25 -d- 2048
144 2013-01-14 12:59:25 -d- 2048
145 2013-01-14 12:59:25 -d- 2048
146 2013-01-14 12:59:25 -d- 2048
147 2013-01-14 12:59:28 -d- 2048
148 2013-01-14 12:59:28 -d- 2048
149 2013-01-14 12:59:28 -d- 2048
150 2013-01-14 12:59:28 -d- 2048
151 2013-01-14 12:59:28 -d- 2048
152 2013-01-14 12:59:28 -d- 2048
153 2013-01-14 12:59:28 -d- 2048
154 2013-01-14 12:59:29 -d- 2048
155 2013-01-14 12:59:32 -d- 2048
156 2013-01-14 12:59:32 -d- 2048
157 2013-01-14 12:59:32 -d- 2048
158 2013-01-14 12:59:32 -d- 2048
159 2013-01-14 12:59:32 -d- 2048
160 2013-01-14 12:59:32 -d- 2048
161 2013-01-14 12:59:32 -d- 2048
162 2013-01-14 12:59:32 -d- 2048
163 2013-01-14 12:59:35 -d- 2048
164 2013-01-14 12:59:35 -d- 2048
165 2013-01-14 12:59:35 -d- 2048
166 2013-01-14 12:59:38 -d- 2048
167 2013-01-14 12:59:38 -d- 2048
168 2013-01-14 12:59:38 -d- 2048
169 2013-01-14 12:59:38 -d- 2048
170 2013-01-14 12:59:38 -d- 2048
171 2013-01-14 12:59:38 -d- 2048
172 2013-01-14 12:59:38 -d- 2048
173 2013-01-14 12:59:38 -d- 2048
174 2013-01-14 12:59:42 -d- 2048
175 2013-01-14 12:59:42 -d- 2048
176 2013-01-14 12:59:42 -d- 2048
177 2013-01-14 12:59:42 -d- 2048
178 2013-01-14 12:59:42 -d- 2048
179 2013-01-14 12:59:42 -d- 2048
180 2013-01-14 12:59:42 -d- 2048
181 2013-01-14 12:59:42 -d- 2048
182 2013-01-14 12:59:45 -d- 2048
183 2013-01-14 12:59:45 -d- 2048
184 2013-01-14 12:59:45 -d- 2048
185 2013-01-14 12:59:45 -d- 2048
186 2013-01-14 12:59:45 -d- 2048
187 2013-01-14 12:59:46 -d- 2048
188 2013-01-14 12:59:46 -d- 2048
189 2013-01-14 12:59:46 -d- 2048
190 2013-01-14 12:59:49 -d- 2048
191 2013-01-14 12:59:49 -d- 2047
192 2013-01-14 12:59:49 -d- 2048
193 2013-01-14 12:59:49 -d- 2048
194 2013-01-14 12:59:49 -d- 2048
195 2013-01-14 12:59:49 -d- 2048
196 2013-01-14 12:59:49 -d- 2048
197 2013-01-14 12:59:49 -d- 2048
198 2013-01-14 12:59:52 -d- 2048
199 2013-01-14 12:59:52 -d- 2048
200 2013-01-14 12:59:52 -d- 2048
201 2013-01-14 12:59:52 -d- 2048
202 2013-01-14 12:59:52 -d- 2048
203 2013-01-14 12:59:52 -d- 2048
204 2013-01-14 12:59:52 -d- 2048
205 2013-01-14 12:59:52 -d- 2048
206 2013-01-14 12:59:56 -d- 2048
207 2013-01-14 12:59:56 -d- 2048
208 2013-01-14 12:59:56 -d- 2048
209 2013-01-14 12:59:56 -d- 2048
210 2013-01-14 12:59:56 -d- 2048
211 2013-01-14 12:59:56 -d- 2048
212 2013-01-14 12:59:56 -d- 2048
213 2013-01-14 12:59:56 -d- 2048
214 2013-01-14 12:59:59 -d- 2048
215 2013-01-14 12:59:59 -d- 2048
216 2013-01-14 12:59:59 -d- 2048
217 2013-01-14 12:59:59 -d- 2048
218 2013-01-14 12:59:59 -d- 2048
219 2013-01-14 12:59:59 -d- 2048
220 2013-01-14 12:59:59 -d- 2048
221 2013-01-14 12:59:59 -d- 2048
222 2013-01-14 13:00:03 -d- 2048
223 2013-01-14 13:00:03 -d- 2048
224 2013-01-14 13:00:03 -d- 2048
225 2013-01-14 13:00:03 -d- 2048
226 2013-01-14 13:00:03 -d- 2048
227 2013-01-14 13:00:03 -d- 2048
228 2013-01-14 13:00:03 -d- 2048
229 2013-01-14 13:00:03 -d- 2048
230 2013-01-14 13:00:06 -d- 2048
231 2013-01-14 13:00:06 -d- 2048
232 2013-01-14 13:00:06 -d- 2048
233 2013-01-14 13:00:06 -d- 2048
234 2013-01-14 13:00:06 -d- 2048
235 2013-01-14 13:00:06 -d- 2048
236 2013-01-14 13:00:06 -d- 2048
237 2013-01-14 13:00:06 -d- 2048
238 2013-01-14 13:00:08 -d- 2048
239 2013-01-14 13:00:08 -d- 2048
240 2013-01-14 13:00:10 -d- 2048
241 2013-01-14 13:00:10 -d- 2048
242 ---------- --:--:-- --- 0
243 2013-01-14 13:00:10 -d- 2048
244 2013-01-14 13:00:10 -d- 2048
245 2013-01-14 13:00:10 -d- 2048
246 2013-01-14 13:00:10 -d- 2048
247 2013-01-14 13:00:10 -d- 2048
248 2013-01-14 13:00:10 -d- 2048
249 2013-01-14 13:00:14 -d- 2048
250 2013-01-14 13:00:14 -d- 2048
251 2013-01-14 13:00:14 -d- 2048
252 2013-01-14 13:00:14 -d- 2048
253 2013-01-14 13:00:14 -d- 2048
254 2013-01-14 13:00:14 -d- 2048
255 2013-01-14 13:00:14 -d- 2048
256 2013-01-14 13:00:14 -d- 2048
257 2013-01-14 13:00:17 -d- 2048
258 2013-01-14 13:00:17 -d- 2048
259 2013-01-14 13:00:17 -d- 2048
260 2013-01-14 13:00:17 -d- 2048
261 2013-01-14 13:00:17 -d- 2048
262 2013-01-14 13:00:17 -d- 2048
263 2013-01-14 13:00:17 -d- 2048
264 2013-01-14 13:00:17 -d- 2048
265 2013-01-14 13:00:21 -d- 2048
266 2013-01-14 13:00:21 -d- 2048
267 2013-01-14 13:00:21 -d- 2048
268 2013-01-14 13:00:21 -d- 2048
269 2013-01-14 13:00:21 -d- 2048
270 2013-01-14 13:00:21 -d- 2048
271 2013-01-14 13:00:21 -d- 2048
272 2013-01-14 13:00:21 -d- 2048
273 2013-01-14 13:00:23 -d- 2048
274 2013-01-14 13:00:23 -d- 2048
275 2013-01-14 13:00:24 ad- 402
276 ---------- --:--:-- ad- 0
277 ---------- --:--:-- --- 0
278 ---------- --:--:-- --- 0
279 ---------- --:--:-- --- 0
280 ---------- --:--:-- --- 0
281 ---------- --:--:-- --- 0
282 ---------- --:--:-- --- 0
283 ---------- --:--:-- --- 0
284 ---------- --:--:-- --- 0
285 ---------- --:--:-- --- 0
286 ---------- --:--:-- --- 0
287 ---------- --:--:-- --- 0
288 ---------- --:--:-- --- 0
289 ---------- --:--:-- --- 0
290 ---------- --:--:-- --- 0
291 ---------- --:--:-- --- 0
292 ---------- --:--:-- --- 0
293 ---------- --:--:-- --- 0
294 ---------- --:--:-- --- 0
295 ---------- --:--:-- --- 0
296 2013-01-14 12:57:46 -d- 2048
297 2013-01-14 12:57:49 -d- 2048
298 2013-01-14 12:57:49 -d- 2048
299 2013-01-14 12:57:49 -d- 2048
300 2013-01-14 12:57:49 -d- 2048
301 2013-01-14 12:57:49 -d- 2048
302 2013-01-14 12:57:49 -d- 2048
303 2013-01-14 12:57:49 -d- 2048
304 2013-01-14 12:57:49 -d- 2048
305 2013-01-14 12:57:49 -d- 2048
306 2013-01-14 12:57:51 -d- 2048
307 2013-01-14 12:57:51 -d- 2048
308 2013-01-14 12:57:51 -d- 2048
309 2013-01-14 12:57:51 -d- 2048
310 2013-01-14 12:57:51 -d- 2048
311 2013-01-14 12:57:51 -d- 2048
312 2013-01-14 12:57:51 -d- 2048
313 2013-01-14 12:57:51 -d- 2048
314 2013-01-14 12:57:53 -d- 2048
315 2013-01-14 12:57:54 -d- 2048
316 2013-01-14 12:57:54 -d- 2048
317 2013-01-14 12:57:55 -d- 2048
318 2013-01-14 12:57:55 -d- 2048
319 2013-01-14 12:57:55 -d- 2048
320 2013-01-14 12:57:55 -d- 2048
321 2013-01-14 12:57:55 -d- 2048
322 2013-01-14 12:57:55 -d- 2048
323 2013-01-14 12:57:58 -d- 2048
324 2013-01-14 12:57:58 -d- 2048
325 2013-01-14 12:57:58 -d- 2048
326 2013-01-14 12:57:58 -d- 2048
327 2013-01-14 12:57:58 -d- 2048
328 2013-01-14 12:57:58 -d- 2048
329 2013-01-14 12:57:58 -d- 2048
330 2013-01-14 12:58:01 -d- 2048
331 2013-01-14 12:58:01 -d- 2048
332 2013-01-14 12:58:01 -d- 2048
333 2013-01-14 12:58:01 -d- 2048
334 2013-01-14 12:58:01 -d- 2048
335 2013-01-14 12:58:01 -d- 2048
336 2013-01-14 12:58:01 -d- 2048
337 2013-01-14 12:58:01 -d- 2048
338 2013-01-14 12:58:05 -d- 2048
339 2013-01-14 12:58:05 -d- 2048
340 2013-01-14 12:58:05 -d- 2048
341 2013-01-14 12:58:05 -d- 2048
342 2013-01-14 12:58:05 -d- 2048
343 2013-01-14 12:58:05 -d- 2048
344 2013-01-14 12:58:05 -d- 2048
345 2013-01-14 12:58:05 -d- 2048
346 2013-01-14 12:58:05 -d- 2048
347 2013-01-14 12:58:08 -d- 2048
348 2013-01-14 12:58:08 -d- 2048
349 2013-01-14 12:58:08 -d- 2048
350 2013-01-14 12:58:08 -d- 2048
351 2013-01-14 12:58:08 -d- 2048
352 2013-01-14 12:58:08 -d- 2048
353 2013-01-14 12:58:08 -d- 2048
354 2013-01-14 12:58:12 -d- 2048
355 2013-01-14 12:58:12 -d- 2048
356 2013-01-14 12:58:12 -d- 2048
357 2013-01-14 12:58:12 -d- 2048
358 2013-01-14 12:58:12 -d- 2048
359 2013-01-14 12:58:12 -d- 2048
360 2013-01-14 12:58:12 -d- 2048
361 2013-01-14 12:58:12 -d- 2048
362 2013-01-14 12:58:12 -d- 2048
363 2013-01-14 12:58:15 -d- 2048
364 2013-01-14 12:58:15 -d- 2048
365 2013-01-14 12:58:15 -d- 2048
366 2013-01-14 12:58:16 -d- 2048
367 2013-01-14 12:58:16 -d- 2048
368 2013-01-14 12:58:16 -d- 2048
369 2013-01-14 12:58:16 -d- 2048
370 2013-01-14 12:58:19 -d- 2048
371 2013-01-14 12:58:19 -d- 2048
372 2013-01-14 12:58:19 -d- 2048
373 2013-01-14 12:58:19 -d- 2048
374 2013-01-14 12:58:19 -d- 2048
375 2013-01-14 12:58:19 -d- 2048
376 2013-01-14 12:58:19 -d- 2048
377 2013-01-14 12:58:19 -d- 2048
378 2013-01-14 12:58:22 -d- 2048
379 2013-01-14 12:58:22 -d- 2048
380 2013-01-14 12:58:22 -d- 2048
381 2013-01-14 12:58:22 -d- 2048
382 2013-01-14 12:58:23 -d- 2048
$ /usr/bin/lscp -abg
CNO DATE TIME MODE FLG NBLKINC ICNT
2369 2013-01-14 12:57:49 cp - 4024 6681
2370 2013-01-14 12:57:53 cp - 292 6682
2371 2013-01-14 12:57:55 cp i 2097 6682
2372 2013-01-14 12:57:58 cp i 2037 6682
2373 2013-01-14 12:58:01 cp i 2029 6682
2374 2013-01-14 12:58:05 cp i 2175 6682
2375 2013-01-14 12:58:08 cp i 1985 6682
2376 2013-01-14 12:58:12 cp i 2110 6682
2377 2013-01-14 12:58:16 cp i 1867 6682
2378 2013-01-14 12:58:19 cp i 1742 6682
2379 2013-01-14 12:58:23 cp i 2030 6682
2380 2013-01-14 12:58:26 cp i 1796 6682
2381 2013-01-14 12:58:29 cp i 1627 6682
2382 2013-01-14 12:58:33 cp i 1579 6682
2383 2013-01-14 12:58:36 cp i 1591 6682
2384 2013-01-14 12:58:40 cp i 1713 6682
2385 2013-01-14 12:58:44 cp i 1894 6682
2386 2013-01-14 12:58:47 cp i 1678 6682
2387 2013-01-14 12:58:51 cp i 1814 6682
2388 2013-01-14 12:58:54 cp i 1635 6682
2389 2013-01-14 12:58:58 cp i 1595 6682
2390 2013-01-14 12:59:01 cp i 1437 6682
2391 2013-01-14 12:59:04 cp i 1328 6682
2392 2013-01-14 12:59:08 cp i 1045 6682
2393 2013-01-14 12:59:12 cp i 680 6682
2394 2013-01-14 12:59:15 cp i 554 6682
2395 2013-01-14 12:59:18 cp i 676 6682
2396 2013-01-14 12:59:22 cp i 626 6682
2397 2013-01-14 12:59:27 cp i 769 6682
2398 2013-01-14 12:59:29 cp i 598 6682
2399 2013-01-14 12:59:35 cp i 748 6682
2400 2013-01-14 12:59:35 cp i 538 6682
2401 2013-01-14 12:59:38 cp i 662 6682
2402 2013-01-14 12:59:42 cp i 620 6682
2403 2013-01-14 12:59:46 cp i 577 6682
2404 2013-01-14 12:59:49 cp i 548 6682
2405 2013-01-14 12:59:52 cp i 458 6682
2406 2013-01-14 12:59:58 cp i 446 6682
2407 2013-01-14 13:00:02 cp i 634 6682
2408 2013-01-14 13:00:03 cp i 588 6682
2409 2013-01-14 13:00:08 cp - 4344 6546
2410 2013-01-14 13:00:10 cp i 2180 6546
2411 2013-01-14 13:00:14 cp i 2173 6546
2412 2013-01-14 13:00:20 cp i 276 6546
2413 2013-01-14 13:00:23 cp i 374 6546
2414 2013-01-14 13:00:24 cp i 402 6546
Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
---
sbin/cleanerd/cleanerd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
index bfcd893..12ed975 100644
--- a/sbin/cleanerd/cleanerd.c
+++ b/sbin/cleanerd/cleanerd.c
@@ -592,7 +592,7 @@ nilfs_cleanerd_select_segments(struct nilfs_cleanerd *cleanerd,
* selected. */
thr = (config->cf_selection_policy.p_threshold != 0) ?
config->cf_selection_policy.p_threshold :
- sustat->ss_nongc_ctime;
+ ~0ULL;
for (segnum = 0; segnum < sustat->ss_nsegs; segnum += n) {
count = (sustat->ss_nsegs - segnum < NILFS_CLEANERD_NSUINFO) ?
--
1.7.10.4
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
[not found] ` <37A149BD-4D8E-4C55-9301-428034C5B9DA-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org>
@ 2013-01-14 18:55 ` Sven Eckelmann
0 siblings, 0 replies; 14+ messages in thread
From: Sven Eckelmann @ 2013-01-14 18:55 UTC (permalink / raw)
To: Vyacheslav Dubeyko; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 3067 bytes --]
On Monday 14 January 2013 22:49:50 Vyacheslav Dubeyko wrote:
> Hi Sven,
>
> On Jan 14, 2013, at 6:54 PM, Sven Eckelmann wrote:
> > The filesystem can end up in a state were the filesystem is full and the
> > returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
> > segments. The garbage collector will not clean anything and therefore no
> > new room for new files will be available and ss_nongc_ctime/sui_lastmod
> > will not be updated without using special tools. This makes the
> > filesystem unusable without manual recovery.
> >
> > Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> > --
> > This problem appeared on a current 3.2 stable kernel (Debian Wheezy
> > build). I am not an FS developer and have therefore not much background
> > knowledge about the NILFS codebase. Nevertheless, this problem hit me
> > quite hard after creating some files on a nilfs partition until it was
> > full and deleting them again.
> >
> > $ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
> > $ rm foo*
> >
> > Looking at the output debugging output using
> >
> > $ watch -n .5 'df -h;tail /var/log/syslog;'
>
> As I understand, you did such sequence of actions:
> 1. Generate such count of files that the whole size of used space was about
> the partition size. 2. Delete all generated files.
> 3. Expect that previously used space is become fully free.
>
> Am I correct?
Nearly. Here is my config file:
protection_period 0
min_clean_segments 10%
max_clean_segments 20%
clean_check_interval 5
selection_policy timestamp
nsegments_per_clean 8
mc_nsegments_per_clean 8
cleaning_interval 5
mc_cleaning_interval 1
retry_interval 40
use_mmap
log_priority debug
and an alternative version
protection_period 20
min_clean_segments 10%
max_clean_segments 20%
clean_check_interval 5
selection_policy timestamp
nsegments_per_clean 8
mc_nsegments_per_clean 8
cleaning_interval 5
mc_cleaning_interval 1
retry_interval 40
use_mmap
log_priority debug
So I would have expected that nilfs would have some free space again after 20
seconds (+ time it needs to free stuff). But it did not and I've showed the
reason in my explanation (threshold vs. the time stored in each reclaimable
segment).
At least "no free space after a work day" was not the expected result using
this setup.
> I think that you have some misunderstanding of NILFS2 working. The GC of
> NILFS2 works in background and it needs to wait sometime before segments
> will be clean. If you have necessity to begin cleaning immediately then you
> can execute "nilfs-clean -p 0". Did you try to use it?
>
> I think that you try to achieve by this patch the result of "nilfs-clean -p
> 0".
This was default set in the configuration file and therefore I did not have to
set it manually using nilfs-clean -p 0.
> Sorry, if I misunderstand something. Please, correct me if I am wrong.
See above.
Kind regards,
Sven
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
[not found] ` <1358178899-26347-1-git-send-email-sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
@ 2013-01-14 19:49 ` Vyacheslav Dubeyko
[not found] ` <37A149BD-4D8E-4C55-9301-428034C5B9DA-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org>
2013-01-15 7:14 ` Vyacheslav Dubeyko
2013-01-15 13:57 ` Vyacheslav Dubeyko
2 siblings, 1 reply; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-14 19:49 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
Hi Sven,
On Jan 14, 2013, at 6:54 PM, Sven Eckelmann wrote:
> The filesystem can end up in a state were the filesystem is full and the
> returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
> segments. The garbage collector will not clean anything and therefore no new
> room for new files will be available and ss_nongc_ctime/sui_lastmod will not be
> updated without using special tools. This makes the filesystem unusable without
> manual recovery.
>
> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> --
> This problem appeared on a current 3.2 stable kernel (Debian Wheezy build). I
> am not an FS developer and have therefore not much background knowledge about
> the NILFS codebase. Nevertheless, this problem hit me quite hard after creating
> some files on a nilfs partition until it was full and deleting them again.
>
> $ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
> $ rm foo*
>
> Looking at the output debugging output using
>
> $ watch -n .5 'df -h;tail /var/log/syslog;'
>
As I understand, you did such sequence of actions:
1. Generate such count of files that the whole size of used space was about the partition size.
2. Delete all generated files.
3. Expect that previously used space is become fully free.
Am I correct?
I think that you have some misunderstanding of NILFS2 working. The GC of NILFS2 works in background and it needs to wait sometime before segments will be clean. If you have necessity to begin cleaning immediately then you can execute "nilfs-clean -p 0". Did you try to use it?
I think that you try to achieve by this patch the result of "nilfs-clean -p 0".
Sorry, if I misunderstand something. Please, correct me if I am wrong.
Thanks,
Vyacheslav Dubeyko.
> clearly showed that it was not finding any segments to delete. The only problem
> I could find was the threshold. After "removing" this threshold, I was able to
> get some clear segments again. I personally cannot explain why the check is
> there at all. Maybe there is a good reason but the comment above it didn't help
> much.
>
> So, here for completeness the threshold: 1358164666 (aka: Mon Jan 14 12:57:46
> CET 2013)
>
> And here are the output of lssu and lscp:
>
> $ lssu --all
> SEGNUM DATE TIME STAT NBLOCKS
> 0 2013-01-14 12:58:23 -d- 2047
> 1 2013-01-14 12:58:23 -d- 2048
> 2 2013-01-14 12:58:23 -d- 2048
> 3 2013-01-14 12:58:26 -d- 2048
> 4 2013-01-14 12:58:26 -d- 2048
> 5 2013-01-14 12:58:26 -d- 2048
> 6 2013-01-14 12:58:26 -d- 2048
> 7 2013-01-14 12:58:26 -d- 2048
> 8 2013-01-14 12:58:26 -d- 2048
> 9 2013-01-14 12:58:26 -d- 2048
> 10 2013-01-14 12:58:26 -d- 2048
> 11 2013-01-14 12:58:29 -d- 2048
> 12 2013-01-14 12:58:29 -d- 2048
> 13 2013-01-14 12:58:29 -d- 2048
> 14 2013-01-14 12:58:29 -d- 2048
> 15 2013-01-14 12:58:29 -d- 2048
> 16 2013-01-14 12:58:29 -d- 2048
> 17 2013-01-14 12:58:29 -d- 2048
> 18 2013-01-14 12:58:29 -d- 2048
> 19 2013-01-14 12:58:32 -d- 2048
> 20 2013-01-14 12:58:32 -d- 2048
> 21 2013-01-14 12:58:32 -d- 2048
> 22 2013-01-14 12:58:33 -d- 2048
> 23 2013-01-14 12:58:33 -d- 2048
> 24 2013-01-14 12:58:33 -d- 2048
> 25 2013-01-14 12:58:33 -d- 2048
> 26 2013-01-14 12:58:33 -d- 2048
> 27 2013-01-14 12:58:36 -d- 2048
> 28 2013-01-14 12:58:36 -d- 2048
> 29 2013-01-14 12:58:36 -d- 2048
> 30 2013-01-14 12:58:36 -d- 2048
> 31 2013-01-14 12:58:36 -d- 2048
> 32 2013-01-14 12:58:36 -d- 2048
> 33 2013-01-14 12:58:36 -d- 2048
> 34 2013-01-14 12:58:36 -d- 2048
> 35 2013-01-14 12:58:40 -d- 2048
> 36 2013-01-14 12:58:40 -d- 2048
> 37 2013-01-14 12:58:40 -d- 2048
> 38 2013-01-14 12:58:40 -d- 2048
> 39 2013-01-14 12:58:40 -d- 2048
> 40 2013-01-14 12:58:40 -d- 2048
> 41 2013-01-14 12:58:40 -d- 2048
> 42 2013-01-14 12:58:40 -d- 2048
> 43 2013-01-14 12:58:43 -d- 2048
> 44 2013-01-14 12:58:43 -d- 2048
> 45 2013-01-14 12:58:43 -d- 2048
> 46 2013-01-14 12:58:43 -d- 2048
> 47 2013-01-14 12:58:43 -d- 2048
> 48 2013-01-14 12:58:43 -d- 2048
> 49 2013-01-14 12:58:43 -d- 2048
> 50 2013-01-14 12:58:43 -d- 2048
> 51 2013-01-14 12:58:47 -d- 2048
> 52 2013-01-14 12:58:47 -d- 2048
> 53 2013-01-14 12:58:47 -d- 2048
> 54 2013-01-14 12:58:47 -d- 2048
> 55 2013-01-14 12:58:47 -d- 2048
> 56 2013-01-14 12:58:47 -d- 2048
> 57 2013-01-14 12:58:47 -d- 2048
> 58 2013-01-14 12:58:47 -d- 2048
> 59 2013-01-14 12:58:50 -d- 2048
> 60 2013-01-14 12:58:51 -d- 2048
> 61 2013-01-14 12:58:51 -d- 2048
> 62 2013-01-14 12:58:51 -d- 2048
> 63 2013-01-14 12:58:51 -d- 2048
> 64 2013-01-14 12:58:51 -d- 2048
> 65 2013-01-14 12:58:51 -d- 2048
> 66 2013-01-14 12:58:51 -d- 2048
> 67 2013-01-14 12:58:54 -d- 2048
> 68 2013-01-14 12:58:54 -d- 2048
> 69 2013-01-14 12:58:54 -d- 2048
> 70 2013-01-14 12:58:54 -d- 2048
> 71 2013-01-14 12:58:54 -d- 2048
> 72 2013-01-14 12:58:54 -d- 2048
> 73 2013-01-14 12:58:54 -d- 2048
> 74 2013-01-14 12:58:54 -d- 2048
> 75 2013-01-14 12:58:57 -d- 2047
> 76 2013-01-14 12:58:57 -d- 2048
> 77 2013-01-14 12:58:57 -d- 2048
> 78 2013-01-14 12:58:57 -d- 2048
> 79 2013-01-14 12:58:57 -d- 2048
> 80 2013-01-14 12:58:57 -d- 2048
> 81 2013-01-14 12:58:57 -d- 2048
> 82 2013-01-14 12:58:57 -d- 2048
> 83 2013-01-14 12:59:01 -d- 2048
> 84 2013-01-14 12:59:01 -d- 2048
> 85 2013-01-14 12:59:01 -d- 2048
> 86 2013-01-14 12:59:01 -d- 2048
> 87 2013-01-14 12:59:01 -d- 2048
> 88 2013-01-14 12:59:01 -d- 2048
> 89 2013-01-14 12:59:01 -d- 2048
> 90 2013-01-14 12:59:01 -d- 2048
> 91 2013-01-14 12:59:04 -d- 2048
> 92 2013-01-14 12:59:04 -d- 2048
> 93 2013-01-14 12:59:04 -d- 2048
> 94 2013-01-14 12:59:04 -d- 2048
> 95 2013-01-14 12:59:04 -d- 2048
> 96 2013-01-14 12:59:04 -d- 2048
> 97 2013-01-14 12:59:04 -d- 2048
> 98 2013-01-14 12:59:04 -d- 2048
> 99 2013-01-14 12:59:08 -d- 2048
> 100 2013-01-14 12:59:08 -d- 2048
> 101 2013-01-14 12:59:08 -d- 2048
> 102 2013-01-14 12:59:08 -d- 2048
> 103 2013-01-14 12:59:08 -d- 2048
> 104 2013-01-14 12:59:08 -d- 2048
> 105 2013-01-14 12:59:08 -d- 2048
> 106 2013-01-14 12:59:08 -d- 2048
> 107 2013-01-14 12:59:11 -d- 2048
> 108 2013-01-14 12:59:11 -d- 2048
> 109 2013-01-14 12:59:11 -d- 2048
> 110 2013-01-14 12:59:11 -d- 2048
> 111 2013-01-14 12:59:11 -d- 2048
> 112 2013-01-14 12:59:12 -d- 2048
> 113 2013-01-14 12:59:12 -d- 2048
> 114 2013-01-14 12:59:12 -d- 2048
> 115 2013-01-14 12:59:15 -d- 2048
> 116 2013-01-14 12:59:15 -d- 2048
> 117 2013-01-14 12:59:15 -d- 2048
> 118 2013-01-14 12:59:15 -d- 2048
> 119 2013-01-14 12:59:15 -d- 2048
> 120 2013-01-14 12:59:15 -d- 2048
> 121 2013-01-14 12:59:15 -d- 2048
> 122 2013-01-14 12:59:15 -d- 2048
> 123 2013-01-14 12:59:18 -d- 2048
> 124 2013-01-14 12:59:18 -d- 2048
> 125 2013-01-14 12:59:18 -d- 2048
> 126 2013-01-14 12:59:18 -d- 2048
> 127 2013-01-14 12:59:18 -d- 2048
> 128 2013-01-14 12:59:18 -d- 2048
> 129 2013-01-14 12:59:18 -d- 2048
> 130 2013-01-14 12:59:18 -d- 2048
> 131 2013-01-14 12:59:21 -d- 2048
> 132 2013-01-14 12:59:21 -d- 2048
> 133 2013-01-14 12:59:21 -d- 2048
> 134 2013-01-14 12:59:21 -d- 2048
> 135 2013-01-14 12:59:21 -d- 2048
> 136 2013-01-14 12:59:21 -d- 2048
> 137 2013-01-14 12:59:22 -d- 2048
> 138 2013-01-14 12:59:22 -d- 2048
> 139 2013-01-14 12:59:25 -d- 2048
> 140 2013-01-14 12:59:25 -d- 2048
> 141 2013-01-14 12:59:25 -d- 2048
> 142 2013-01-14 12:59:25 -d- 2048
> 143 2013-01-14 12:59:25 -d- 2048
> 144 2013-01-14 12:59:25 -d- 2048
> 145 2013-01-14 12:59:25 -d- 2048
> 146 2013-01-14 12:59:25 -d- 2048
> 147 2013-01-14 12:59:28 -d- 2048
> 148 2013-01-14 12:59:28 -d- 2048
> 149 2013-01-14 12:59:28 -d- 2048
> 150 2013-01-14 12:59:28 -d- 2048
> 151 2013-01-14 12:59:28 -d- 2048
> 152 2013-01-14 12:59:28 -d- 2048
> 153 2013-01-14 12:59:28 -d- 2048
> 154 2013-01-14 12:59:29 -d- 2048
> 155 2013-01-14 12:59:32 -d- 2048
> 156 2013-01-14 12:59:32 -d- 2048
> 157 2013-01-14 12:59:32 -d- 2048
> 158 2013-01-14 12:59:32 -d- 2048
> 159 2013-01-14 12:59:32 -d- 2048
> 160 2013-01-14 12:59:32 -d- 2048
> 161 2013-01-14 12:59:32 -d- 2048
> 162 2013-01-14 12:59:32 -d- 2048
> 163 2013-01-14 12:59:35 -d- 2048
> 164 2013-01-14 12:59:35 -d- 2048
> 165 2013-01-14 12:59:35 -d- 2048
> 166 2013-01-14 12:59:38 -d- 2048
> 167 2013-01-14 12:59:38 -d- 2048
> 168 2013-01-14 12:59:38 -d- 2048
> 169 2013-01-14 12:59:38 -d- 2048
> 170 2013-01-14 12:59:38 -d- 2048
> 171 2013-01-14 12:59:38 -d- 2048
> 172 2013-01-14 12:59:38 -d- 2048
> 173 2013-01-14 12:59:38 -d- 2048
> 174 2013-01-14 12:59:42 -d- 2048
> 175 2013-01-14 12:59:42 -d- 2048
> 176 2013-01-14 12:59:42 -d- 2048
> 177 2013-01-14 12:59:42 -d- 2048
> 178 2013-01-14 12:59:42 -d- 2048
> 179 2013-01-14 12:59:42 -d- 2048
> 180 2013-01-14 12:59:42 -d- 2048
> 181 2013-01-14 12:59:42 -d- 2048
> 182 2013-01-14 12:59:45 -d- 2048
> 183 2013-01-14 12:59:45 -d- 2048
> 184 2013-01-14 12:59:45 -d- 2048
> 185 2013-01-14 12:59:45 -d- 2048
> 186 2013-01-14 12:59:45 -d- 2048
> 187 2013-01-14 12:59:46 -d- 2048
> 188 2013-01-14 12:59:46 -d- 2048
> 189 2013-01-14 12:59:46 -d- 2048
> 190 2013-01-14 12:59:49 -d- 2048
> 191 2013-01-14 12:59:49 -d- 2047
> 192 2013-01-14 12:59:49 -d- 2048
> 193 2013-01-14 12:59:49 -d- 2048
> 194 2013-01-14 12:59:49 -d- 2048
> 195 2013-01-14 12:59:49 -d- 2048
> 196 2013-01-14 12:59:49 -d- 2048
> 197 2013-01-14 12:59:49 -d- 2048
> 198 2013-01-14 12:59:52 -d- 2048
> 199 2013-01-14 12:59:52 -d- 2048
> 200 2013-01-14 12:59:52 -d- 2048
> 201 2013-01-14 12:59:52 -d- 2048
> 202 2013-01-14 12:59:52 -d- 2048
> 203 2013-01-14 12:59:52 -d- 2048
> 204 2013-01-14 12:59:52 -d- 2048
> 205 2013-01-14 12:59:52 -d- 2048
> 206 2013-01-14 12:59:56 -d- 2048
> 207 2013-01-14 12:59:56 -d- 2048
> 208 2013-01-14 12:59:56 -d- 2048
> 209 2013-01-14 12:59:56 -d- 2048
> 210 2013-01-14 12:59:56 -d- 2048
> 211 2013-01-14 12:59:56 -d- 2048
> 212 2013-01-14 12:59:56 -d- 2048
> 213 2013-01-14 12:59:56 -d- 2048
> 214 2013-01-14 12:59:59 -d- 2048
> 215 2013-01-14 12:59:59 -d- 2048
> 216 2013-01-14 12:59:59 -d- 2048
> 217 2013-01-14 12:59:59 -d- 2048
> 218 2013-01-14 12:59:59 -d- 2048
> 219 2013-01-14 12:59:59 -d- 2048
> 220 2013-01-14 12:59:59 -d- 2048
> 221 2013-01-14 12:59:59 -d- 2048
> 222 2013-01-14 13:00:03 -d- 2048
> 223 2013-01-14 13:00:03 -d- 2048
> 224 2013-01-14 13:00:03 -d- 2048
> 225 2013-01-14 13:00:03 -d- 2048
> 226 2013-01-14 13:00:03 -d- 2048
> 227 2013-01-14 13:00:03 -d- 2048
> 228 2013-01-14 13:00:03 -d- 2048
> 229 2013-01-14 13:00:03 -d- 2048
> 230 2013-01-14 13:00:06 -d- 2048
> 231 2013-01-14 13:00:06 -d- 2048
> 232 2013-01-14 13:00:06 -d- 2048
> 233 2013-01-14 13:00:06 -d- 2048
> 234 2013-01-14 13:00:06 -d- 2048
> 235 2013-01-14 13:00:06 -d- 2048
> 236 2013-01-14 13:00:06 -d- 2048
> 237 2013-01-14 13:00:06 -d- 2048
> 238 2013-01-14 13:00:08 -d- 2048
> 239 2013-01-14 13:00:08 -d- 2048
> 240 2013-01-14 13:00:10 -d- 2048
> 241 2013-01-14 13:00:10 -d- 2048
> 242 ---------- --:--:-- --- 0
> 243 2013-01-14 13:00:10 -d- 2048
> 244 2013-01-14 13:00:10 -d- 2048
> 245 2013-01-14 13:00:10 -d- 2048
> 246 2013-01-14 13:00:10 -d- 2048
> 247 2013-01-14 13:00:10 -d- 2048
> 248 2013-01-14 13:00:10 -d- 2048
> 249 2013-01-14 13:00:14 -d- 2048
> 250 2013-01-14 13:00:14 -d- 2048
> 251 2013-01-14 13:00:14 -d- 2048
> 252 2013-01-14 13:00:14 -d- 2048
> 253 2013-01-14 13:00:14 -d- 2048
> 254 2013-01-14 13:00:14 -d- 2048
> 255 2013-01-14 13:00:14 -d- 2048
> 256 2013-01-14 13:00:14 -d- 2048
> 257 2013-01-14 13:00:17 -d- 2048
> 258 2013-01-14 13:00:17 -d- 2048
> 259 2013-01-14 13:00:17 -d- 2048
> 260 2013-01-14 13:00:17 -d- 2048
> 261 2013-01-14 13:00:17 -d- 2048
> 262 2013-01-14 13:00:17 -d- 2048
> 263 2013-01-14 13:00:17 -d- 2048
> 264 2013-01-14 13:00:17 -d- 2048
> 265 2013-01-14 13:00:21 -d- 2048
> 266 2013-01-14 13:00:21 -d- 2048
> 267 2013-01-14 13:00:21 -d- 2048
> 268 2013-01-14 13:00:21 -d- 2048
> 269 2013-01-14 13:00:21 -d- 2048
> 270 2013-01-14 13:00:21 -d- 2048
> 271 2013-01-14 13:00:21 -d- 2048
> 272 2013-01-14 13:00:21 -d- 2048
> 273 2013-01-14 13:00:23 -d- 2048
> 274 2013-01-14 13:00:23 -d- 2048
> 275 2013-01-14 13:00:24 ad- 402
> 276 ---------- --:--:-- ad- 0
> 277 ---------- --:--:-- --- 0
> 278 ---------- --:--:-- --- 0
> 279 ---------- --:--:-- --- 0
> 280 ---------- --:--:-- --- 0
> 281 ---------- --:--:-- --- 0
> 282 ---------- --:--:-- --- 0
> 283 ---------- --:--:-- --- 0
> 284 ---------- --:--:-- --- 0
> 285 ---------- --:--:-- --- 0
> 286 ---------- --:--:-- --- 0
> 287 ---------- --:--:-- --- 0
> 288 ---------- --:--:-- --- 0
> 289 ---------- --:--:-- --- 0
> 290 ---------- --:--:-- --- 0
> 291 ---------- --:--:-- --- 0
> 292 ---------- --:--:-- --- 0
> 293 ---------- --:--:-- --- 0
> 294 ---------- --:--:-- --- 0
> 295 ---------- --:--:-- --- 0
> 296 2013-01-14 12:57:46 -d- 2048
> 297 2013-01-14 12:57:49 -d- 2048
> 298 2013-01-14 12:57:49 -d- 2048
> 299 2013-01-14 12:57:49 -d- 2048
> 300 2013-01-14 12:57:49 -d- 2048
> 301 2013-01-14 12:57:49 -d- 2048
> 302 2013-01-14 12:57:49 -d- 2048
> 303 2013-01-14 12:57:49 -d- 2048
> 304 2013-01-14 12:57:49 -d- 2048
> 305 2013-01-14 12:57:49 -d- 2048
> 306 2013-01-14 12:57:51 -d- 2048
> 307 2013-01-14 12:57:51 -d- 2048
> 308 2013-01-14 12:57:51 -d- 2048
> 309 2013-01-14 12:57:51 -d- 2048
> 310 2013-01-14 12:57:51 -d- 2048
> 311 2013-01-14 12:57:51 -d- 2048
> 312 2013-01-14 12:57:51 -d- 2048
> 313 2013-01-14 12:57:51 -d- 2048
> 314 2013-01-14 12:57:53 -d- 2048
> 315 2013-01-14 12:57:54 -d- 2048
> 316 2013-01-14 12:57:54 -d- 2048
> 317 2013-01-14 12:57:55 -d- 2048
> 318 2013-01-14 12:57:55 -d- 2048
> 319 2013-01-14 12:57:55 -d- 2048
> 320 2013-01-14 12:57:55 -d- 2048
> 321 2013-01-14 12:57:55 -d- 2048
> 322 2013-01-14 12:57:55 -d- 2048
> 323 2013-01-14 12:57:58 -d- 2048
> 324 2013-01-14 12:57:58 -d- 2048
> 325 2013-01-14 12:57:58 -d- 2048
> 326 2013-01-14 12:57:58 -d- 2048
> 327 2013-01-14 12:57:58 -d- 2048
> 328 2013-01-14 12:57:58 -d- 2048
> 329 2013-01-14 12:57:58 -d- 2048
> 330 2013-01-14 12:58:01 -d- 2048
> 331 2013-01-14 12:58:01 -d- 2048
> 332 2013-01-14 12:58:01 -d- 2048
> 333 2013-01-14 12:58:01 -d- 2048
> 334 2013-01-14 12:58:01 -d- 2048
> 335 2013-01-14 12:58:01 -d- 2048
> 336 2013-01-14 12:58:01 -d- 2048
> 337 2013-01-14 12:58:01 -d- 2048
> 338 2013-01-14 12:58:05 -d- 2048
> 339 2013-01-14 12:58:05 -d- 2048
> 340 2013-01-14 12:58:05 -d- 2048
> 341 2013-01-14 12:58:05 -d- 2048
> 342 2013-01-14 12:58:05 -d- 2048
> 343 2013-01-14 12:58:05 -d- 2048
> 344 2013-01-14 12:58:05 -d- 2048
> 345 2013-01-14 12:58:05 -d- 2048
> 346 2013-01-14 12:58:05 -d- 2048
> 347 2013-01-14 12:58:08 -d- 2048
> 348 2013-01-14 12:58:08 -d- 2048
> 349 2013-01-14 12:58:08 -d- 2048
> 350 2013-01-14 12:58:08 -d- 2048
> 351 2013-01-14 12:58:08 -d- 2048
> 352 2013-01-14 12:58:08 -d- 2048
> 353 2013-01-14 12:58:08 -d- 2048
> 354 2013-01-14 12:58:12 -d- 2048
> 355 2013-01-14 12:58:12 -d- 2048
> 356 2013-01-14 12:58:12 -d- 2048
> 357 2013-01-14 12:58:12 -d- 2048
> 358 2013-01-14 12:58:12 -d- 2048
> 359 2013-01-14 12:58:12 -d- 2048
> 360 2013-01-14 12:58:12 -d- 2048
> 361 2013-01-14 12:58:12 -d- 2048
> 362 2013-01-14 12:58:12 -d- 2048
> 363 2013-01-14 12:58:15 -d- 2048
> 364 2013-01-14 12:58:15 -d- 2048
> 365 2013-01-14 12:58:15 -d- 2048
> 366 2013-01-14 12:58:16 -d- 2048
> 367 2013-01-14 12:58:16 -d- 2048
> 368 2013-01-14 12:58:16 -d- 2048
> 369 2013-01-14 12:58:16 -d- 2048
> 370 2013-01-14 12:58:19 -d- 2048
> 371 2013-01-14 12:58:19 -d- 2048
> 372 2013-01-14 12:58:19 -d- 2048
> 373 2013-01-14 12:58:19 -d- 2048
> 374 2013-01-14 12:58:19 -d- 2048
> 375 2013-01-14 12:58:19 -d- 2048
> 376 2013-01-14 12:58:19 -d- 2048
> 377 2013-01-14 12:58:19 -d- 2048
> 378 2013-01-14 12:58:22 -d- 2048
> 379 2013-01-14 12:58:22 -d- 2048
> 380 2013-01-14 12:58:22 -d- 2048
> 381 2013-01-14 12:58:22 -d- 2048
> 382 2013-01-14 12:58:23 -d- 2048
>
> $ /usr/bin/lscp -abg
> CNO DATE TIME MODE FLG NBLKINC ICNT
> 2369 2013-01-14 12:57:49 cp - 4024 6681
> 2370 2013-01-14 12:57:53 cp - 292 6682
> 2371 2013-01-14 12:57:55 cp i 2097 6682
> 2372 2013-01-14 12:57:58 cp i 2037 6682
> 2373 2013-01-14 12:58:01 cp i 2029 6682
> 2374 2013-01-14 12:58:05 cp i 2175 6682
> 2375 2013-01-14 12:58:08 cp i 1985 6682
> 2376 2013-01-14 12:58:12 cp i 2110 6682
> 2377 2013-01-14 12:58:16 cp i 1867 6682
> 2378 2013-01-14 12:58:19 cp i 1742 6682
> 2379 2013-01-14 12:58:23 cp i 2030 6682
> 2380 2013-01-14 12:58:26 cp i 1796 6682
> 2381 2013-01-14 12:58:29 cp i 1627 6682
> 2382 2013-01-14 12:58:33 cp i 1579 6682
> 2383 2013-01-14 12:58:36 cp i 1591 6682
> 2384 2013-01-14 12:58:40 cp i 1713 6682
> 2385 2013-01-14 12:58:44 cp i 1894 6682
> 2386 2013-01-14 12:58:47 cp i 1678 6682
> 2387 2013-01-14 12:58:51 cp i 1814 6682
> 2388 2013-01-14 12:58:54 cp i 1635 6682
> 2389 2013-01-14 12:58:58 cp i 1595 6682
> 2390 2013-01-14 12:59:01 cp i 1437 6682
> 2391 2013-01-14 12:59:04 cp i 1328 6682
> 2392 2013-01-14 12:59:08 cp i 1045 6682
> 2393 2013-01-14 12:59:12 cp i 680 6682
> 2394 2013-01-14 12:59:15 cp i 554 6682
> 2395 2013-01-14 12:59:18 cp i 676 6682
> 2396 2013-01-14 12:59:22 cp i 626 6682
> 2397 2013-01-14 12:59:27 cp i 769 6682
> 2398 2013-01-14 12:59:29 cp i 598 6682
> 2399 2013-01-14 12:59:35 cp i 748 6682
> 2400 2013-01-14 12:59:35 cp i 538 6682
> 2401 2013-01-14 12:59:38 cp i 662 6682
> 2402 2013-01-14 12:59:42 cp i 620 6682
> 2403 2013-01-14 12:59:46 cp i 577 6682
> 2404 2013-01-14 12:59:49 cp i 548 6682
> 2405 2013-01-14 12:59:52 cp i 458 6682
> 2406 2013-01-14 12:59:58 cp i 446 6682
> 2407 2013-01-14 13:00:02 cp i 634 6682
> 2408 2013-01-14 13:00:03 cp i 588 6682
> 2409 2013-01-14 13:00:08 cp - 4344 6546
> 2410 2013-01-14 13:00:10 cp i 2180 6546
> 2411 2013-01-14 13:00:14 cp i 2173 6546
> 2412 2013-01-14 13:00:20 cp i 276 6546
> 2413 2013-01-14 13:00:23 cp i 374 6546
> 2414 2013-01-14 13:00:24 cp i 402 6546
>
> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> ---
> sbin/cleanerd/cleanerd.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index bfcd893..12ed975 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -592,7 +592,7 @@ nilfs_cleanerd_select_segments(struct nilfs_cleanerd *cleanerd,
> * selected. */
> thr = (config->cf_selection_policy.p_threshold != 0) ?
> config->cf_selection_policy.p_threshold :
> - sustat->ss_nongc_ctime;
> + ~0ULL;
>
> for (segnum = 0; segnum < sustat->ss_nsegs; segnum += n) {
> count = (sustat->ss_nsegs - segnum < NILFS_CLEANERD_NSUINFO) ?
> --
> 1.7.10.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
[not found] ` <1358178899-26347-1-git-send-email-sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
2013-01-14 19:49 ` Vyacheslav Dubeyko
@ 2013-01-15 7:14 ` Vyacheslav Dubeyko
2013-01-15 8:34 ` Sven Eckelmann
2013-01-15 10:43 ` Ryusuke Konishi
2013-01-15 13:57 ` Vyacheslav Dubeyko
2 siblings, 2 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-15 7:14 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Mon, 2013-01-14 at 16:54 +0100, Sven Eckelmann wrote:
> The filesystem can end up in a state were the filesystem is full and the
> returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
> segments. The garbage collector will not clean anything and therefore no new
> room for new files will be available and ss_nongc_ctime/sui_lastmod will not be
> updated without using special tools. This makes the filesystem unusable without
> manual recovery.
>
> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> --
> This problem appeared on a current 3.2 stable kernel (Debian Wheezy build). I
> am not an FS developer and have therefore not much background knowledge about
> the NILFS codebase. Nevertheless, this problem hit me quite hard after creating
> some files on a nilfs partition until it was full and deleting them again.
>
> $ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
> $ rm foo*
>
> Looking at the output debugging output using
>
> $ watch -n .5 'df -h;tail /var/log/syslog;'
>
> clearly showed that it was not finding any segments to delete. The only problem
> I could find was the threshold. After "removing" this threshold, I was able to
> get some clear segments again. I personally cannot explain why the check is
> there at all. Maybe there is a good reason but the comment above it didn't help
> much.
>
> So, here for completeness the threshold: 1358164666 (aka: Mon Jan 14 12:57:46
> CET 2013)
>
> And here are the output of lssu and lscp:
>
> $ lssu --all
> SEGNUM DATE TIME STAT NBLOCKS
> 0 2013-01-14 12:58:23 -d- 2047
> 1 2013-01-14 12:58:23 -d- 2048
[snip]
>
> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> ---
> sbin/cleanerd/cleanerd.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> index bfcd893..12ed975 100644
> --- a/sbin/cleanerd/cleanerd.c
> +++ b/sbin/cleanerd/cleanerd.c
> @@ -592,7 +592,7 @@ nilfs_cleanerd_select_segments(struct nilfs_cleanerd *cleanerd,
> * selected. */
> thr = (config->cf_selection_policy.p_threshold != 0) ?
> config->cf_selection_policy.p_threshold :
> - sustat->ss_nongc_ctime;
> + ~0ULL;
>
As I understand the code of nilfs_cleanerd, this code is correct without
your changing. The ss_nongc_ctime is the creation time of the last
segment not for GC. When thr is set then it compared with sui_lastmod.
The sui_lastmod is the timestamp of last modification. So, the
nilfs_cleanerd works right.
I think that this is a bug on the kernel side. My current vision is that
in some environment the ns_nongc_ctime can be not updated correctly. So,
you have such threshold that prevent from segments clearing.
Thank you for the issue report.
With the best regards,
Vyacheslav Dubeyko.
> for (segnum = 0; segnum < sustat->ss_nsegs; segnum += n) {
> count = (sustat->ss_nsegs - segnum < NILFS_CLEANERD_NSUINFO) ?
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-15 7:14 ` Vyacheslav Dubeyko
@ 2013-01-15 8:34 ` Sven Eckelmann
2013-01-15 10:43 ` Ryusuke Konishi
1 sibling, 0 replies; 14+ messages in thread
From: Sven Eckelmann @ 2013-01-15 8:34 UTC (permalink / raw)
To: Vyacheslav Dubeyko; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]
On Tuesday 15 January 2013 11:14:17 Vyacheslav Dubeyko wrote:
[...]
> > Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> > ---
> >
> > sbin/cleanerd/cleanerd.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
> > index bfcd893..12ed975 100644
> > --- a/sbin/cleanerd/cleanerd.c
> > +++ b/sbin/cleanerd/cleanerd.c
> > @@ -592,7 +592,7 @@ nilfs_cleanerd_select_segments(struct nilfs_cleanerd
> > *cleanerd,>
> > * selected. */
> >
> > thr = (config->cf_selection_policy.p_threshold != 0) ?
> >
> > config->cf_selection_policy.p_threshold :
> > - sustat->ss_nongc_ctime;
> > + ~0ULL;
>
> As I understand the code of nilfs_cleanerd, this code is correct without
> your changing. The ss_nongc_ctime is the creation time of the last
> segment not for GC. When thr is set then it compared with sui_lastmod.
> The sui_lastmod is the timestamp of last modification. So, the
> nilfs_cleanerd works right.
Depends on the definition of right. In a complete environment using the current
nilfs kernel code it needs to _work around_ this problem. But yes, the culprit
must be the kernel side.
> I think that this is a bug on the kernel side. My current vision is that
> in some environment the ns_nongc_ctime can be not updated correctly. So,
> you have such threshold that prevent from segments clearing.
Yes, one problem for example is the non-monotonic behaviour of many timekeeper
functions.
Kind regards,
Sven
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-15 7:14 ` Vyacheslav Dubeyko
2013-01-15 8:34 ` Sven Eckelmann
@ 2013-01-15 10:43 ` Ryusuke Konishi
[not found] ` <201301151043.AA04492-ZdTO5nnmHvkOizVVqyxoihMFgDP4sedm@public.gmane.org>
1 sibling, 1 reply; 14+ messages in thread
From: Ryusuke Konishi @ 2013-01-15 10:43 UTC (permalink / raw)
To: Vyacheslav Dubeyko; +Cc: Sven Eckelmann, linux-nilfs-u79uwXL29TY76Z2rM5mHXA
Hi Vyacheslav,
>On Mon, 2013-01-14 at 16:54 +0100, Sven Eckelmann wrote:
>> The filesystem can end up in a state were the filesystem is full and the
>> returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
>> segments. The garbage collector will not clean anything and therefore no new
>> room for new files will be available and ss_nongc_ctime/sui_lastmod will not be
>> updated without using special tools. This makes the filesystem unusable without
>> manual recovery.
>>
>> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
>> --
>> This problem appeared on a current 3.2 stable kernel (Debian Wheezy build). I
>> am not an FS developer and have therefore not much background knowledge about
>> the NILFS codebase. Nevertheless, this problem hit me quite hard after creating
>> some files on a nilfs partition until it was full and deleting them again.
>>
>> $ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
>> $ rm foo*
>>
>> Looking at the output debugging output using
>>
>> $ watch -n .5 'df -h;tail /var/log/syslog;'
>>
>> clearly showed that it was not finding any segments to delete. The only problem
>> I could find was the threshold. After "removing" this threshold, I was able to
>> get some clear segments again. I personally cannot explain why the check is
>> there at all. Maybe there is a good reason but the comment above it didn't help
>> much.
>>
>> So, here for completeness the threshold: 1358164666 (aka: Mon Jan 14 12:57:46
>> CET 2013)
>>
>> And here are the output of lssu and lscp:
>>
>> $ lssu --all
>> SEGNUM DATE TIME STAT NBLOCKS
>> 0 2013-01-14 12:58:23 -d- 2047
>> 1 2013-01-14 12:58:23 -d- 2048
>
>[snip]
>
>>
>> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
>> ---
>> sbin/cleanerd/cleanerd.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/sbin/cleanerd/cleanerd.c b/sbin/cleanerd/cleanerd.c
>> index bfcd893..12ed975 100644
>> --- a/sbin/cleanerd/cleanerd.c
>> +++ b/sbin/cleanerd/cleanerd.c
>> @@ -592,7 +592,7 @@ nilfs_cleanerd_select_segments(struct nilfs_cleanerd *cleanerd,
>> * selected. */
>> thr = (config->cf_selection_policy.p_threshold != 0) ?
>> config->cf_selection_policy.p_threshold :
>> - sustat->ss_nongc_ctime;
>> + ~0ULL;
>>
>
>As I understand the code of nilfs_cleanerd, this code is correct without
>your changing. The ss_nongc_ctime is the creation time of the last
>segment not for GC. When thr is set then it compared with sui_lastmod.
>The sui_lastmod is the timestamp of last modification. So, the
>nilfs_cleanerd works right.
>
>I think that this is a bug on the kernel side. My current vision is that
>in some environment the ns_nongc_ctime can be not updated correctly. So,
>you have such threshold that prevent from segments clearing.
I guess the problem is caused because ss_nongc_ctime is not updated by
the kernel module if dirty blocks created by file operations are written
out through nilfs_clean_segments function. Actually,
nilfs_clean_segments() can roll together normal dirty blocks.
So, if GC is working without interruption, ss_nongc_ctime may not be
updated long time.
If this is the cause of the problem, nilfs2 log writer should be changed
so that nilfs->ns_nongc_ctime is updated with sci->sc_seg_ctime if
there is a file system change (file/directory change or an ifile change).
NILFS_SC_HAVE_DELTA flag may be available for this purpose.
The ss_nongc_ctime check is needed to make GC stoppable.
Without this check, GC can continue endlessly even if the file system
doesn't have any change.
With regards,
Ryusuke Konishi
>Thank you for the issue report.
>
>With the best regards,
>Vyacheslav Dubeyko.
>
>> for (segnum = 0; segnum < sustat->ss_nsegs; segnum += n) {
>> count = (sustat->ss_nsegs - segnum < NILFS_CLEANERD_NSUINFO) ?
>
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
>the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
[not found] ` <201301151043.AA04492-ZdTO5nnmHvkOizVVqyxoihMFgDP4sedm@public.gmane.org>
@ 2013-01-15 13:48 ` Vyacheslav Dubeyko
0 siblings, 0 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-15 13:48 UTC (permalink / raw)
To: Ryusuke Konishi; +Cc: Sven Eckelmann, linux-nilfs-u79uwXL29TY76Z2rM5mHXA
Hi Ryusuke,
On Tue, 2013-01-15 at 19:43 +0900, Ryusuke Konishi wrote:
[snip]
> >
> >As I understand the code of nilfs_cleanerd, this code is correct without
> >your changing. The ss_nongc_ctime is the creation time of the last
> >segment not for GC. When thr is set then it compared with sui_lastmod.
> >The sui_lastmod is the timestamp of last modification. So, the
> >nilfs_cleanerd works right.
> >
> >I think that this is a bug on the kernel side. My current vision is that
> >in some environment the ns_nongc_ctime can be not updated correctly. So,
> >you have such threshold that prevent from segments clearing.
>
> I guess the problem is caused because ss_nongc_ctime is not updated by
> the kernel module if dirty blocks created by file operations are written
> out through nilfs_clean_segments function. Actually,
> nilfs_clean_segments() can roll together normal dirty blocks.
>
> So, if GC is working without interruption, ss_nongc_ctime may not be
> updated long time.
>
> If this is the cause of the problem, nilfs2 log writer should be changed
> so that nilfs->ns_nongc_ctime is updated with sci->sc_seg_ctime if
> there is a file system change (file/directory change or an ifile change).
>
> NILFS_SC_HAVE_DELTA flag may be available for this purpose.
>
> The ss_nongc_ctime check is needed to make GC stoppable.
> Without this check, GC can continue endlessly even if the file system
> doesn't have any change.
>
Thank you for your detailed explanation. It is very helpful for me.
But, currently, I can't reproduce the issue. All works fine on my side.
With the best regards,
Vyacheslav Dubeyko.
> With regards,
> Ryusuke Konishi
>
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
[not found] ` <1358178899-26347-1-git-send-email-sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
2013-01-14 19:49 ` Vyacheslav Dubeyko
2013-01-15 7:14 ` Vyacheslav Dubeyko
@ 2013-01-15 13:57 ` Vyacheslav Dubeyko
2013-01-15 15:17 ` Sven Eckelmann
2 siblings, 1 reply; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-15 13:57 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Mon, 2013-01-14 at 16:54 +0100, Sven Eckelmann wrote:
> The filesystem can end up in a state were the filesystem is full and the
> returned ss_nongc_ctime is smaller than sui_lastmod of all reclaimable
> segments. The garbage collector will not clean anything and therefore no new
> room for new files will be available and ss_nongc_ctime/sui_lastmod will not be
> updated without using special tools. This makes the filesystem unusable without
> manual recovery.
>
> Signed-off-by: Sven Eckelmann <sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
> --
> This problem appeared on a current 3.2 stable kernel (Debian Wheezy build). I
> am not an FS developer and have therefore not much background knowledge about
> the NILFS codebase. Nevertheless, this problem hit me quite hard after creating
> some files on a nilfs partition until it was full and deleting them again.
>
> $ for i in `seq 0 150`; do dd if=/dev/zero of=foo$i count=22528; done
> $ rm foo*
>
> Looking at the output debugging output using
>
> $ watch -n .5 'df -h;tail /var/log/syslog;'
>
> clearly showed that it was not finding any segments to delete. The only problem
> I could find was the threshold. After "removing" this threshold, I was able to
> get some clear segments again. I personally cannot explain why the check is
> there at all. Maybe there is a good reason but the comment above it didn't help
> much.
>
> So, here for completeness the threshold: 1358164666 (aka: Mon Jan 14 12:57:46
> CET 2013)
>
Unfortunately, currently, I can't reproduce the issue. All works fine on
my side.
Could you share more details about your environment? What version of
nilfs-utils do you use?
Maybe do you have some NILFS2-related error messages in your system log?
Or, maybe, reproducing path is more complex as you described?
With the best regards,
Vyacheslav Dubeyko.
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-15 13:57 ` Vyacheslav Dubeyko
@ 2013-01-15 15:17 ` Sven Eckelmann
2013-01-17 7:57 ` Vyacheslav Dubeyko
0 siblings, 1 reply; 14+ messages in thread
From: Sven Eckelmann @ 2013-01-15 15:17 UTC (permalink / raw)
To: Vyacheslav Dubeyko; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]
On Tuesday 15 January 2013 17:57:21 Vyacheslav Dubeyko wrote:
> Could you share more details about your environment? What version of
> nilfs-utils do you use?
>
> Maybe do you have some NILFS2-related error messages in your system log?
>
> Or, maybe, reproducing path is more complex as you described?
System running Debian Wheezy (amd64) with linux 3.2.35-2 and nilfs-tools
2.1.4-1. The nilfs partition is a secondary partition on a CFCard and is not
used in normal situations. There are different processes running in the
background... for example ntpd. There are no interesting infos in the system
log (unless you are interested in nilfs_cleanerd output saying nothing else
than "pause"/"sleep"/"0 segments selected to be cleaned"/...).
Following way is a good way to force a similar problem (this is not the way I
produced it the first time, but it is a possible scenario):
$ cd /nilfs/
$ ntpdate-debian
$ date -s @`date +'%s'|awk '{ print $1+9000}'`
$ for i in `seq 0 300`; do dd if=/dev/zero of=foo$i count=22528; done
$ sleep 360
$ ntpdate-debian
$ touch asd
$ rm *
$ date -s @`date +'%s'|awk '{ print $1+10000}'`
$ mount -o remount /nilfs/
It is a little bit harsh, but a similar thing can easily happen in real world
with non-monotonic clocks (but the jumps are usually a little smaller).
Kind regards,
Sven
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-15 15:17 ` Sven Eckelmann
@ 2013-01-17 7:57 ` Vyacheslav Dubeyko
2013-01-17 8:32 ` Sven Eckelmann
2013-01-18 6:17 ` Vyacheslav Dubeyko
0 siblings, 2 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-17 7:57 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Tue, 2013-01-15 at 16:17 +0100, Sven Eckelmann wrote:
> On Tuesday 15 January 2013 17:57:21 Vyacheslav Dubeyko wrote:
> > Could you share more details about your environment? What version of
> > nilfs-utils do you use?
> >
> > Maybe do you have some NILFS2-related error messages in your system log?
> >
> > Or, maybe, reproducing path is more complex as you described?
>
> System running Debian Wheezy (amd64) with linux 3.2.35-2 and nilfs-tools
> 2.1.4-1. The nilfs partition is a secondary partition on a CFCard and is not
> used in normal situations. There are different processes running in the
> background... for example ntpd. There are no interesting infos in the system
> log (unless you are interested in nilfs_cleanerd output saying nothing else
> than "pause"/"sleep"/"0 segments selected to be cleaned"/...).
>
> Following way is a good way to force a similar problem (this is not the way I
> produced it the first time, but it is a possible scenario):
>
> $ cd /nilfs/
> $ ntpdate-debian
> $ date -s @`date +'%s'|awk '{ print $1+9000}'`
> $ for i in `seq 0 300`; do dd if=/dev/zero of=foo$i count=22528; done
> $ sleep 360
> $ ntpdate-debian
> $ touch asd
> $ rm *
> $ date -s @`date +'%s'|awk '{ print $1+10000}'`
> $ mount -o remount /nilfs/
>
> It is a little bit harsh, but a similar thing can easily happen in real world
> with non-monotonic clocks (but the jumps are usually a little smaller).
>
Yes, I can reproduce the issue. But I think that it doesn't make sense
to change something in timestamp policy of GC. Unfortunately, we have
currently only timestamp policy. And I think that it needs to use
another GC policy in the case of such issue. So, the proper fix will be
adding another GC policy.
Anyway, I think that possible time deviation can achieve about minutes
in real life. And probability to encounter such issue in real life very
low. I think so because it is hardly to encounter in real life of
coincidence such factors as (1) to fill a whole volume and then to
remove all files during several minutes and (2) immediately update
system time.
If you do so by the hands then, from my point of view, you know what you
are doing.
As a resume, I think that proper fix of such issue can be adding of
additional GC policies that is not affected by such use-case.
Thanks,
Vyacheslav Dubeyko.
> Kind regards,
> Sven
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-17 7:57 ` Vyacheslav Dubeyko
@ 2013-01-17 8:32 ` Sven Eckelmann
2013-01-17 9:19 ` Vyacheslav Dubeyko
2013-01-18 7:04 ` Vyacheslav Dubeyko
2013-01-18 6:17 ` Vyacheslav Dubeyko
1 sibling, 2 replies; 14+ messages in thread
From: Sven Eckelmann @ 2013-01-17 8:32 UTC (permalink / raw)
To: Vyacheslav Dubeyko; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
[-- Attachment #1: Type: text/plain, Size: 337 bytes --]
On Thursday 17 January 2013 11:57:18 Vyacheslav Dubeyko wrote:
[...]
> If you do so by the hands then, from my point of view, you know what you
> are doing.
You asked for a way to easily reproduce it and now say it is my fault? Sry,
but it happened in a real world scenario and I only provided a testcase for
you.
Kind regards,
Sven
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-17 8:32 ` Sven Eckelmann
@ 2013-01-17 9:19 ` Vyacheslav Dubeyko
2013-01-18 7:04 ` Vyacheslav Dubeyko
1 sibling, 0 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-17 9:19 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Thu, 2013-01-17 at 09:32 +0100, Sven Eckelmann wrote:
> On Thursday 17 January 2013 11:57:18 Vyacheslav Dubeyko wrote:
> [...]
> > If you do so by the hands then, from my point of view, you know what you
> > are doing.
>
> You asked for a way to easily reproduce it and now say it is my fault? Sry,
> but it happened in a real world scenario and I only provided a testcase for
> you.
>
No, I don't say that it is your fault. The timestamp policy of GC has
some limitation and peculiarities because of nature of the time. So, you
described one of such use-case. But, from my point of view, such
use-case has very low probability of occurrence. And proper fix can be
adding of another GC policy. That's all I said.
With the best regards,
Vyacheslav Dubeyko.
> Kind regards,
> Sven
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-17 7:57 ` Vyacheslav Dubeyko
2013-01-17 8:32 ` Sven Eckelmann
@ 2013-01-18 6:17 ` Vyacheslav Dubeyko
1 sibling, 0 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-18 6:17 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Thu, 2013-01-17 at 11:57 +0400, Vyacheslav Dubeyko wrote:
> On Tue, 2013-01-15 at 16:17 +0100, Sven Eckelmann wrote:
> > On Tuesday 15 January 2013 17:57:21 Vyacheslav Dubeyko wrote:
> > > Could you share more details about your environment? What version of
> > > nilfs-utils do you use?
> > >
> > > Maybe do you have some NILFS2-related error messages in your system log?
> > >
> > > Or, maybe, reproducing path is more complex as you described?
> >
> > System running Debian Wheezy (amd64) with linux 3.2.35-2 and nilfs-tools
> > 2.1.4-1. The nilfs partition is a secondary partition on a CFCard and is not
> > used in normal situations. There are different processes running in the
> > background... for example ntpd. There are no interesting infos in the system
> > log (unless you are interested in nilfs_cleanerd output saying nothing else
> > than "pause"/"sleep"/"0 segments selected to be cleaned"/...).
> >
> > Following way is a good way to force a similar problem (this is not the way I
> > produced it the first time, but it is a possible scenario):
> >
> > $ cd /nilfs/
> > $ ntpdate-debian
> > $ date -s @`date +'%s'|awk '{ print $1+9000}'`
> > $ for i in `seq 0 300`; do dd if=/dev/zero of=foo$i count=22528; done
> > $ sleep 360
> > $ ntpdate-debian
> > $ touch asd
> > $ rm *
> > $ date -s @`date +'%s'|awk '{ print $1+10000}'`
> > $ mount -o remount /nilfs/
> >
> > It is a little bit harsh, but a similar thing can easily happen in real world
> > with non-monotonic clocks (but the jumps are usually a little smaller).
> >
>
> Yes, I can reproduce the issue. But I think that it doesn't make sense
> to change something in timestamp policy of GC. Unfortunately, we have
> currently only timestamp policy. And I think that it needs to use
> another GC policy in the case of such issue. So, the proper fix will be
> adding another GC policy.
>
> Anyway, I think that possible time deviation can achieve about minutes
> in real life. And probability to encounter such issue in real life very
> low. I think so because it is hardly to encounter in real life of
> coincidence such factors as (1) to fill a whole volume and then to
> remove all files during several minutes and (2) immediately update
> system time.
>
> If you do so by the hands then, from my point of view, you know what you
> are doing.
>
> As a resume, I think that proper fix of such issue can be adding of
> additional GC policies that is not affected by such use-case.
>
After additional thinking about the issue, I am thinking about
nilfs-clean utility modification. I mean that it needs to have
opportunity to define time point of the last write time (ss_nongc_ctime)
by hands in the case of the issue occurrence. So, an user should have
opportunity to detect situation when volume hasn't free segments because
of GC idleness and to correct GC's behaviour by hands (likewise starting
GC by "nilfs-clean -p 0").
With the best regards,
Vyacheslav Dubeyko.
> Thanks,
> Vyacheslav Dubeyko.
>
> > Kind regards,
> > Sven
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] nilfs-utils: Work around uncleanable full filesystem
2013-01-17 8:32 ` Sven Eckelmann
2013-01-17 9:19 ` Vyacheslav Dubeyko
@ 2013-01-18 7:04 ` Vyacheslav Dubeyko
1 sibling, 0 replies; 14+ messages in thread
From: Vyacheslav Dubeyko @ 2013-01-18 7:04 UTC (permalink / raw)
To: Sven Eckelmann; +Cc: linux-nilfs-u79uwXL29TY76Z2rM5mHXA
On Thu, 2013-01-17 at 11:57 +0400, Vyacheslav Dubeyko wrote:
> On Tue, 2013-01-15 at 16:17 +0100, Sven Eckelmann wrote:
> > On Tuesday 15 January 2013 17:57:21 Vyacheslav Dubeyko wrote:
> > > Could you share more details about your environment? What version of
> > > nilfs-utils do you use?
> > >
> > > Maybe do you have some NILFS2-related error messages in your system log?
> > >
> > > Or, maybe, reproducing path is more complex as you described?
> >
> > System running Debian Wheezy (amd64) with linux 3.2.35-2 and nilfs-tools
> > 2.1.4-1. The nilfs partition is a secondary partition on a CFCard and is not
> > used in normal situations. There are different processes running in the
> > background... for example ntpd. There are no interesting infos in the system
> > log (unless you are interested in nilfs_cleanerd output saying nothing else
> > than "pause"/"sleep"/"0 segments selected to be cleaned"/...).
> >
> > Following way is a good way to force a similar problem (this is not the way I
> > produced it the first time, but it is a possible scenario):
> >
> > $ cd /nilfs/
> > $ ntpdate-debian
> > $ date -s @`date +'%s'|awk '{ print $1+9000}'`
> > $ for i in `seq 0 300`; do dd if=/dev/zero of=foo$i count=22528; done
> > $ sleep 360
> > $ ntpdate-debian
> > $ touch asd
> > $ rm *
> > $ date -s @`date +'%s'|awk '{ print $1+10000}'`
> > $ mount -o remount /nilfs/
> >
> > It is a little bit harsh, but a similar thing can easily happen in real world
> > with non-monotonic clocks (but the jumps are usually a little smaller).
> >
>
> Yes, I can reproduce the issue. But I think that it doesn't make sense
> to change something in timestamp policy of GC. Unfortunately, we have
> currently only timestamp policy. And I think that it needs to use
> another GC policy in the case of such issue. So, the proper fix will be
> adding another GC policy.
>
> Anyway, I think that possible time deviation can achieve about minutes
> in real life. And probability to encounter such issue in real life very
> low. I think so because it is hardly to encounter in real life of
> coincidence such factors as (1) to fill a whole volume and then to
> remove all files during several minutes and (2) immediately update
> system time.
>
> If you do so by the hands then, from my point of view, you know what you
> are doing.
>
> As a resume, I think that proper fix of such issue can be adding of
> additional GC policies that is not affected by such use-case.
>
After additional thinking about the issue, I am thinking about
nilfs-clean utility modification. I mean that it needs to have
opportunity to define time point of the last write time (ss_nongc_ctime)
by hands in the case of the issue occurrence. So, an user should have
opportunity to detect situation when volume hasn't free segments because
of GC idleness and to correct GC's behaviour by hands (likewise starting
GC by "nilfs-clean -p 0").
With the best regards,
Vyacheslav Dubeyko.
> Thanks,
> Vyacheslav Dubeyko.
>
> > Kind regards,
> > Sven
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2013-01-18 7:04 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-14 15:54 [PATCH] nilfs-utils: Work around uncleanable full filesystem Sven Eckelmann
[not found] ` <1358178899-26347-1-git-send-email-sven-KaDOiPu9UxWEi8DpZVb4nw@public.gmane.org>
2013-01-14 19:49 ` Vyacheslav Dubeyko
[not found] ` <37A149BD-4D8E-4C55-9301-428034C5B9DA-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org>
2013-01-14 18:55 ` Sven Eckelmann
2013-01-15 7:14 ` Vyacheslav Dubeyko
2013-01-15 8:34 ` Sven Eckelmann
2013-01-15 10:43 ` Ryusuke Konishi
[not found] ` <201301151043.AA04492-ZdTO5nnmHvkOizVVqyxoihMFgDP4sedm@public.gmane.org>
2013-01-15 13:48 ` Vyacheslav Dubeyko
2013-01-15 13:57 ` Vyacheslav Dubeyko
2013-01-15 15:17 ` Sven Eckelmann
2013-01-17 7:57 ` Vyacheslav Dubeyko
2013-01-17 8:32 ` Sven Eckelmann
2013-01-17 9:19 ` Vyacheslav Dubeyko
2013-01-18 7:04 ` Vyacheslav Dubeyko
2013-01-18 6:17 ` Vyacheslav Dubeyko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox