arachne

Project Page: http://arachne-framework.org/ Kickstarter: https://www.kickstarter.com/projects/1346708779/arachne-rapid-web-development-for-clojure
jcf 2018-01-30T20:18:43.000799Z

The number of files opened by Fileset is tricky to provide. I've upped limits to 8192 and still Arachne's falling over.

jcf 2018-01-30T20:19:24.000395Z

I'm not sure if it's a leak per se, but Figwheel and Boot both build fairly large codebases without requiring the same resources as Fileset.

jcf 2018-01-30T20:20:17.000185Z

Counting open files, and listing a couple of the tmp directories.

luke 2018-01-30T20:20:29.000171Z

@jcf this is after explicitly using arachne-fileset 1.6.1?

luke 2018-01-30T20:20:47.000084Z

1.6.0 definitely had a bug where it did not release file handles

jcf 2018-01-30T20:20:50.000685Z

Yep.

luke 2018-01-30T20:21:28.000495Z

hm, ok

luke 2018-01-30T20:21:59.000587Z

It is intentional that Arachne keep a reference to all files in a fileset that is currently “in use”.

jcf 2018-01-30T20:22:02.000006Z

Just pushed the commit with the pinned version: https://github.com/jcf/usr/commit/6221c464215e90fb963600ba9015327ae2a7fb9c

luke 2018-01-30T20:22:20.000428Z

otherwise the problem was that temp files would get cleaned up out from under Arachne while it was running

luke 2018-01-30T20:22:36.000135Z

but it should only be the “current” set of files actually available via the web server

luke 2018-01-30T20:22:44.000379Z

intermediate filesets should be cleaned up

jcf 2018-01-30T20:23:16.000622Z

I saw this comment, but not sure that's what you're referring to:

;; Idea: we could theoretically do garbage collection, if space in the blob store becomes an issue:
;; - find all instances of FileSet (would require registering in a weak map at creation)
;; - find all TmpFiles in all FileSets
;; - delete all blobs not referenced by a TmpFile
;; - but it's probably unnecessary

luke 2018-01-30T20:23:29.000201Z

I’ll take a look at your project; this might be “expected” if you have more than 8192 files in your public dir.

luke 2018-01-30T20:23:44.000590Z

@jcf no, that’s about file cleanup from disk, not open file pointers

1👍
jcf 2018-01-30T20:25:49.000410Z

Most of the files come from Closure I think. Not me. 🙂

jcf 2018-01-30T20:25:58.000643Z

And other deps.

jcf 2018-01-30T20:26:04.000786Z

Me indirectly.

luke 2018-01-30T20:26:07.000303Z

right, fair enough. I’ll investigate

jcf 2018-01-30T20:26:26.000303Z

I'd like to help. I don't know the fileset code that well, but will have a peruse.

luke 2018-01-30T20:27:08.000253Z

I need to run now, but if you think this is something you could get your head around and want to have a call to get context, I’d be happy to do so.

jcf 2018-01-30T20:28:05.000443Z

A call would certainly help, but in the mean time I'll familiarise myself with Fileset.

jcf 2018-01-30T20:28:27.000246Z

I've not directly used NIO APIs a lot yet so this'll be interesting. 🙂