Clojurians Log v2

i'm seeing an analysis failure with no errors - https://cljdoc.org/builds/38385 - when i try running the analyzer locally

clojure -Sdeps '{:deps {cljdoc/cljdoc-analyzer {:git/url "<https://github.com/cljdoc/cljdoc-analyzer.git>" :sha "9ff24dd10772b5c95d33d1c93c866a9e38884c17"}}}' -m cljdoc-analyzer.main analyze \
  --project cc.qbits/alia-all --version "5.0.0-alpha2-SNAPSHOT" \
  --output-filename "cc-qbits-alia-all-5.0.0-alpha2-SNAPSHOT.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki \
  --extra-repo "clojars <https://repo.clojars.org/>" \
  --extra-repo "central <http://central.maven.org/maven2/>"

I can take a peek @mccraigmccraig…

@mccraigmccraig, First guess: It likely has to do with https://github.com/mpenet/alia’s use of modules. Have you read cljdoc docs on https://github.com/cljdoc/cljdoc/blob/master/doc/userguide/for-library-authors.adoc#module-support?

And.. there might be something fishy with the modules too? If I try to run the analyzer on cc.qbits/alia-joda-time version 5.0.0-alpha1 it seems to be referencing version 5.0.0 of alia which is not up on clojars.

ah, yeah - there was an error in the deps ... it's fixed now, but clearly i need to do another release

ok, that's a couple of things to run with ... thanks for the pointers @lee

https://github.com/metosin/reitit is a great reference for a project with with modules https://cljdoc.org/d/metosin/reitit/0.5.10/doc/introduction. It’s https://github.com/metosin/reitit/blob/master/doc/cljdoc.edn might be of particular interest to you.

Also… don’t forget to make sure scm gets setup appropriately in your pom.xml so that cljdoc can fetch and display your docs.

After addressing those issues, and if you are still running into probs, come on back with more Qs!

awesome, thanks @lee - i found https://github.com/cljdoc/cljdoc/blob/master/doc/running-cljdoc-locally.adoc too, so hopefully that will help

Yeah, I recently reworked that doc, feedback is always appreciated!

should i be able to run an ingest locally on the multi-module project if all the deps are only installed in my local repo @lee? i've added :scm keys to all modules and a doc/cljdoc.edn and i'm getting a very similar failure locally to that i was seeing on cljdoc circleci : https://gist.githubusercontent.com/mccraigmccraig/642198b30a0c8a28e61dc35b1d1d66dd/raw/f526a995eba994154907c1251d7626332cbaea7c/failed%2520local%2520cljdoc%2520alia%2520ingest

I’m not sure I ever tried a module project locally. Lemme check…

i also tried it without the explicit jar/pom paths, and it retrieved them from the local repo ok... updated the gist to show that

I just tried

clojure -m cljdoc-analyzer.main analyze \
  --project metosin/reitit --version "0.5.10" \
  --output-filename "/tmp/erp.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki 

Lessee…

So are you following https://github.com/cljdoc/cljdoc/blob/master/doc/running-cljdoc-locally.adoc#testing-docs-for-a-library-during-development?

yes, exactly

using ../cljdoc/script/cljdoc rather than docker

cool

i'll try ingesting reitit too, as a test of my local setup

good idea

It’s kind of a horrible error message, that we deliver for this scenario! :simple_smile: Have you setup your doc/cljdoc.edn?

The :cljdoc/include-namespaces-from-dependencies would be important for alia

yep, doc/cljdoc.edn is there - https://github.com/mpenet/alia/blob/master/doc/cljdoc.edn

I’m comparing your cljdoc.edn to the one in reitit…

Reitit is listing what look like maven coords, and I think you might be listing namespaces?

ohhhh

:man-facepalming:

It’s not entirely clear from our docs what to list, we should certainly add a tip/note.

clearly i read that key :cljdoc/include-namespaces-from-dependencies and though "aha, this is a list of namespaces" , without looking carefully at what reitit had done

yeah, I can see why you (and others) might do that!

I don’t think there are a ton of projects on cljdoc that use modules, so while you might be going through a bit of pain here, we’ll end up at least improving our docs!

I sense extreme success for alia on cljdoc on the horizon. :simple_smile:

darn, nope. same error 😞

when i ran the analyzer directly on reitit, it is successful

but it still fails on alia, so it is some attribute of the alia project which is causing the issue

oh right… I tried the analyzer rather than ingest on reitit. Did you successfully ingest reitit?

i also tried ingest (on reitit-parent which is the multimodule project) - it seemed to succeed

https://gist.githubusercontent.com/mccraigmccraig/32e4cc696a9286ec08859742e9a798f0/raw/8a773225a4bb828f916bc24fe5437e3bd9923065/reitit%2520ingest%2520run

So you’ve installed all your modules to your local maven repo?

yes

Is everything currently pushed to alia? Maybe I can diagnose. I’ll need the command to install to local repo.

everything is currently pushed. to install, lein sub install ; lein install should do the trick

you could try lein test, but unless you have a cassandra instance on localhost:9042 that will fail... although it will fail after loading namespaces, so you should be able to see that all the submodules were installed correctly

cool, won’t run the tests, but will install and try to see what’s up

thanks @lee

ok… good, I am seeing the same error as you.

phew 🙂

still poking around… starting with one of your modules… which oddly does not seem to analyze either.

don't know if this is relevant - when i ran vanilla codox i had to do some :exclusions messing to get it to run - there were asm clashes between one of codox's deps and com.datastax.oss/java-driver-core-shaded

com.datastax.oss/java-driver-core-shaded is a dep of the "main" module cc.qbits/alia

once i excluded all the asm from com.datastax.oss/java-driver-core-shaded, and let codox have its own preferred version, it worked ok

that might be relevant, but first…

it seems that cljdoc does understand the use of dependencyManagement in a pom.xml. If I explicitly put versions in dependencies I can successfully locally ingest cc.qbits/alia-joda-time

doh! lol, i only put the dependencyManagement stuff in yday - it cleaned up all the project.cljs quite a lot

ok, so i'll have to get rid of the dependencyManagement for the moment then... should i file a ticket against https://github.com/cljdoc/cljdoc-analyzer ?

But… reitit uses :managed-dependencies in its project.clj…

interesting

But always specifies a version on :dependencies ?

So maybe there is some partial advantage to use :managed-dependencies? Not a lein guy, dunno.

i'm looking in pom.xml for metosin/reitit and its not got any version against the <dependencies> , only against the <dependencyManagement>

Huh, right you are…

but...

maybe only for his modules?

all the <dependencies> in the top level metosin/reitit-parent project do seem to have versions

ok, well i think i can add explicit versions for all the top-level deps easily enough

hmmm… I don’t think we’ve figure it out yet.

2020-12-23 18:05:58,931 INFO cljdoc-analyzer.runner - Analysis succeeded.

oh… if I remove the explicit dep on org.clojure/clojure in the pom, I have success for cc.qbits/alia-joda-time

So… I currently think I might have been wrong about dependencyManagement

i've added explicit versions to all the <dependencies> and now i'm passing analysis and ingest seems to be working

ok, I’m not sure of the root cause, but if you are happy, I am happy.

not 100% sure yet - how do i see the results on the local server ? when i search on localhost:8000 , it seems to hit the remote api ?

ah, gotit... http://localhost:8000/d/cc.qbits/alia-all/5.0.0-alpha2-SNAPSHOT/doc/introduction

Yep!

Did your docs import?

although i'm not seeing any namespaces 😞

just the README

goodness gracious

so the import of your modules did not work, I guess.

no, but it crashed less hard than previously

baby steps

i will now see what happens if i put explicit versions in all the module dependencies

from what I see on reitit, that should not be necessary… but…

it's worth a try, it's the only action which has shown any progress at all so far

Ok, FWIW, here’s an observation: If I add an explicit version only to org.clojure/clojure in your poms I am getting success.

At least on the two modules I tried it on.

analysis success, or you can see the API docs ?

ingest success

yeah, i'm getting ingest success, can't see any API docs though

still working on root cause of original failure… There’s something that does not like a versionless org.clojure/clojure dep.

ok, I’ll move on… have you pushed your changes?

not yet...

Not to overwhelm you but you’ll probably want to be more specific with your scm . So that cljdoc can point back to the correct revision on github.

(Unrelated to your current issue.)

np, not overwhelmed yet 🙂

i just pushed changes to add explicit versions everywhere... i also added a scope to all the org.clojure/clojure deps

oh. I see. gotta ingest all modules when testing locally

oh... got to ingest them in dependency order ?

cljdoc is consulting its db for modules at render time.

order is unimportant. Modules that are not ingested won’t show up under namespaces.

In production this would just happen automatically after the first cljdoc load.

oh - w00t - i ingested a module, and re-ran the top-level, and now i see API docs 😃

sweet!

So only known thing left is your scm tag. https://github.com/cljdoc/cljdoc/blob/master/doc/userguide/faq.md#how-do-i-set-scm-info-for-my-project

i already added that - https://github.com/mpenet/alia/blob/master/project.clj#L4

and in all the modules too...

it was coming through to the pom.xml last i looked

hmm… maybe lein does something different for local install than for deploy?

If I look at a reitit pom I see

<scm>
    &lt;url&gt;<https://github.com/metosin/reitit>&lt;/url&gt;
    &lt;connection&gt;scm:git:<git://github.com/metosin/reitit.git&lt;/connection>&gt;
    &lt;developerConnection&gt;scm:git:<ssh://git@github.com/metosin/reitit.git&lt;/developerConnection>&gt;
    &lt;tag&gt;e27d5c02138fd129280bcd1d18c49ac2067f93fc&lt;/tag&gt;
  &lt;/scm&gt;

Oh no wait yours seems fine now, I must have been looking at an earlier version that only had url

cool - yeah, i added it after you first said way back when

ah! good stuff!

so, extreme success?

looking that way ... i'm just seeing how much duplication i can get rid of in the metadata, then i'll try a new release

thank you for all the help @lee - do you need an issue submitting on gh ?

ok, i note that metosin/reitit has no org.clojure/clojure deps at all in its modules - which also avoids the issue - so, i think i'll follow that pattern, rather than have a hard-coded version in each module

So… what did we uncover? 1. docs on :cljdoc/include-namespaces-from-dependencies could use more guidance 2. docs on running locally should mention that you’ll need to ingest all your modules 3. there is something up with having org.clojure/clojure under dependencyManagement and then versionless under dependencies Anything else?

I’ll hit 1 and 2 today. Not sure what to do about 3. Is it a Clojure thing? Or a cljdoc thing?

i thought versionless under dependencies is the point of dependencyManagement - so you can specify a single version once across multiple libs, and i presumed its a maven feature rather than anything specifically clojure

but that's just my impression, i haven't really looked at it beyond reading some docs which suggested it as a way of reducing duplication and inconsistency

Yeah, in maven/java world it works fine. Don’t know root cause. I might poke around a small bit.

it's interesting that the issue is only on the org.clojure/clojure dep - i've just removed the version from another dependency and it seems to analyse fine

weird huh?

the thing i think is a (maybe indirect) cljdoc issue is that there was no useful error message

oh right! ya, so when there were no resolvable deps cljdoc got very mysterious with us.

So if I am right about the indirect cause, an error message like “no dependencies found” might have been helpful.

I think it should be easy to setup a small reproducible project for that.

I can log that one.

yes, something which gives some basis for experimentation - if i had gotten any inkling that it was something to do with the org.clojure/clojure dep and version i would probably have tried removing it and figured out a path to success

@mccraigmccraig, here’s what I came up with for an issue: https://github.com/cljdoc/cljdoc-analyzer/issues/34

yep, that covers it all i think @lee

thanks again for your help - i doubt i would have gotten a solution without it

My pleasure @mccraigmccraig, thanks for uncovering some interesting problems!

@mccraigmccraig docs have been improved/clarified a bit around sub modules: https://github.com/cljdoc/cljdoc/commit/cd97bbf7d763c13949ce6c26ea74d89b1c2ba9ed

i think that covers it @lee