Clojurians Log v2

The number of files opened by Fileset is tricky to provide. I've upped limits to 8192 and still Arachne's falling over.

I'm not sure if it's a leak per se, but Figwheel and Boot both build fairly large codebases without requiring the same resources as Fileset.

Counting open files, and listing a couple of the tmp directories.

@jcf this is after explicitly using arachne-fileset 1.6.1?

1.6.0 definitely had a bug where it did not release file handles

Yep.

hm, ok

It is intentional that Arachne keep a reference to all files in a fileset that is currently “in use”.

Just pushed the commit with the pinned version: https://github.com/jcf/usr/commit/6221c464215e90fb963600ba9015327ae2a7fb9c

otherwise the problem was that temp files would get cleaned up out from under Arachne while it was running

but it should only be the “current” set of files actually available via the web server

intermediate filesets should be cleaned up

I saw this comment, but not sure that's what you're referring to:

;; Idea: we could theoretically do garbage collection, if space in the blob store becomes an issue:
;; - find all instances of FileSet (would require registering in a weak map at creation)
;; - find all TmpFiles in all FileSets
;; - delete all blobs not referenced by a TmpFile
;; - but it's probably unnecessary

I’ll take a look at your project; this might be “expected” if you have more than 8192 files in your public dir.

From here: https://github.com/arachne-framework/arachne-fileset/blob/master/src/arachne/fileset.clj#L12

@jcf no, that’s about file cleanup from disk, not open file pointers

@jcf uploaded a file: https://clojurians.slack.com/files/U06FTAZV3/F926SNZ0F/tree___code_usr.txt

Most of the files come from Closure I think. Not me. 🙂

And other deps.

Me indirectly.

right, fair enough. I’ll investigate

I'd like to help. I don't know the fileset code that well, but will have a peruse.

I need to run now, but if you think this is something you could get your head around and want to have a call to get context, I’d be happy to do so.

A call would certainly help, but in the mean time I'll familiarise myself with Fileset.

I've not directly used NIO APIs a lot yet so this'll be interesting. 🙂