Clojurians Log v2

I just released a prerelease clj 1.10.1.745 with the new dep tree printing under clj -X:deps tree (want to get some feedback but I will extend to -Stree before I release it). Options available (won't be available under -Stree): • :file <path> - make tree from a trace.edn file created by clj -Strace instead of the local deps.edn • :indent <int> - tree spacing (default = 2) • :hide-libs <set-of-libs> - libs to hide in the tree under the root, defaults to #{org.clojure/clojure}

I'm also thinking about having a flag to choose whether to display excluded libs (-Stree does not, the new format does)

anyways, feedback welcome

there are also apis to get as data, might be useful to have a version to print or spit the tree as edn instead of printing? dunno

I could use edn output, then pipe into jet for manipulation.

Hey all. I'm clutching at straws here so apologies for the noise. 🙂 Anyone tried installing deps today and noticed classpath issues like this:

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central (<https://repo1.maven.org/maven2/>)

I started out creating a question here because I thought it was a problem with the Datomic Maven repo but I can download the JAR with aws s3 cp just fine. http://ask.datomic.com/index.php/546/could-not-find-artifact-com-datomic-ion-jar-0-9-48-in-central

This works:

aws s3 cp <s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.48/ion-0.9.48.jar> .

$ clojure -h | head
Version: 1.10.1.739

It works on my machine. 😄

do you have AWS creds set?

On my machine, yep. Defaults and profiles. In the Docker container I had AWS_ACCESS_KEY_ID and secret exported at build time, but have since moved that so the creds are only there at runtime. Inside the GitHub Action I've not exported anything or configured AWS in any way.

Are you thinking my AWS creds are making things work on this host machine?

I read that for Maven repos on S3 there's some AWS API interaction required to get the region. I guess that could be failing.

yes, it needs to determine the bucket location

You can stick a ?region=... on the end of the :mvn/repo URL I think. I could try that.

I don't know which region the Datomic Cloud bucket is in but I should be able to find out.

it's us-east-1

I'll try that now. Thanks Mr. Miller. Uno momento.

Building a Docker container, and running a job (?) via GitHub Actions.

No joy with GitHub:

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: joda-time/joda-time/2.10/joda-time-2.10.jar from central
Downloading: commons-codec/commons-codec/1.15/commons-codec-1.15.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central (<https://repo1.maven.org/maven2/>)

Docker sits here for a few seconds and then continues on its way:

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud

Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Downloading: tigris/tigris/0.1.2/tigris-0.1.2.jar from clojars
Downloading: javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar from central
Downloading: com/cognitect/aws/appsync/809.2.784.0/appsync-809.2.784.0.jar from central
Downloading: com/amazonaws/aws-java-sdk-s3/1.11.210/aws-java-sdk-s3-1.11.210.jar from central
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central (<https://repo1.maven.org/maven2/>)
ERROR: Service 'clojure' failed to build : The command '/bin/sh -c clojure -Srepro -P -A:dev:test:cider-clj' returned a non-zero code: 1

I'm not sure why tools.deps is downloading things twice.

Should there be two lines "Downloading" com.datomic/ion ?

yes, pom and jar

Ahh, that makes sense.

Hmm. I'm still stumped in that case. :thinking_face:

Okay, I can reproduce locally.

mv ~/.aws{,-ignore}
mv ~/.m2/repository/com/datomic/ion{,-ignore}

error in process sentinel: Could not start nREPL server: Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true
DEPRECATED: Libs must be qualified, change compliment => compliment/compliment 
DEPRECATED: Libs must be qualified, change nrepl => nrepl/nrepl 
DEPRECATED: Libs must be qualified, change refactor-nrepl => refactor-nrepl/refactor-nrepl 
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.pom from datomic-cloud
Downloading: org/clojure/clojure/maven-metadata.xml from datomic-cloud
Downloading: com/datomic/ion/0.9.48/ion-0.9.48.jar from datomic-cloud
Error building classpath. Could not find artifact com.datomic:ion:jar:0.9.48 in central (<https://repo1.maven.org/maven2/>)

Interestingly there's nothing in my ~/.m2/repository/com/datomic/ion/0.9.48 directory…

I can see two .part files show up momentarily, and then they disappear after I'm told classpath construction failed.

it has to do with the aws creds of whatever user you're using

it does not have iam access to s3 ops or something like that

So to pull down the Ion dependency I need to configure AWS and ensure I have at least some S3 permissions?

you need to be using aws user credentials that don't prevent s3 use for HeadObject, GetObject, and maybe GetBucketLocation

Okay, I'll create an IAM policy that grants that access. 👍 I can test that pretty quickly with Docker. GitHub Actions will take a little longer. 🙂

Dos momentos.

the objects you're going after definitely still exist and are public in the bucket, the issue here is around IAM on the accessing user

@alexmiller I see you've answered my question over on http://ask.datomic.com. I've added the following policy and I'm seeing the same error:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:GetBucketLocation",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::datomic-releases-1fc2183a/*"
        }
    ]
}

sorry I'm not an IAM guru

is there some kind of simulator or something that can answer whether this user has access to some resource?

I've added the credentials to my local machine and I'm trying out the operations using the AWS CLI.

Copying the JAR worked. Just checking I can fetch the bucket location.

Okay, so I'm not allowed to get the bucket location. That's not surprising though as it's not my bucket. I shouldn't be able to grant myself permission to interact with buckets owned by other people.

$ aws --profile REDACTED s3api get-bucket-location --bucket datomic-releases-1fc2183a

An error occurred (AccessDenied) when calling the GetBucketLocation operation: Access Denied

$ aws --profile REDACTED s3 cp <s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.48/ion-0.9.48.jar> .

download: <s3://datomic-releases-1fc2183a/maven/releases/com/datomic/ion/0.9.48/ion-0.9.48.jar> to ./ion-0.9.48.jar

@alexmiller Any further thoughts on TDEPS-174 and/or TDEPS-175? I'm finding more and more that those are obstacles to things we want to do at work 😞

this bucket has a policy that GetBucketLocation is open to *

you may be hitting something region related though - what's the default region for this user?

buckets are inherently region-specific

this is the case where adding the ?region=us-east-1 suffix to the url should help I think

haven't worked on it yet

Why does clj download this long list of deps with an empty deps.edn? Among these is an AWS api lib. Why? I'll post this as a snippet.

When setting :mvn/local-repo "/tmp/mvn/foo" the list is even longer. I thought the tools jar was an uberjar that contained all clj needs

@borkdude Is it possibly the deps in the system deps.edn?

I guess so yes:

:aliases {
    :deps {:replace-deps {org.clojure/tools.deps.alpha {:mvn/version "0.9.833"}
                          org.slf4j/slf4j-nop {:mvn/version "1.7.25"}}
           :ns-default clojure.tools.cli.api}
    :test {:extra-paths ["test"]}
  }

The question remains: why is there an AWS api lib among the deps of tools.deps.alpha. That feels not right.

fetching from an s3 repo, maybe?

Ah right, that must be it!

The tree looks great. But to get the deps tree of tools.deps I had to add it to the main deps, because you can't activate an alias. So running into the same problem that Sean ran into.

@borkdude I don't fully understand why those aren't part of uberjar either though :)

the :deps alias doesn't use the uberjar

running -X:deps tree is in no way magical and doesn't use any parts of clj - it's just running a program that is in tools.deps.alpha library

the uberjar is used by clj to get the classpath of the program to run

@borkdude I'm a little confused by "But to get the deps tree of tools.deps I had to add it to the main deps, because you can't activate an alias. So running into the same problem that Sean ran into."

or nvm, I think I am reading it properly now

@alexmiller it's not magical, but the deps it downloads are probably already in the uberjar. but I get the point of not using that as that would be magical. unless :deps was by convention a built-in thing that always uses the tools jar.

correct

it's intentionally not that because there is no way to add additional deps and then de-dupe across the uberjar

uberjars are inherently unfriendly to combination

good point

uberjars are evil, but a useful evil if you keep them to yourself

anyways, I'm going to change -Stree to do the format above too, so you'll be able to do all the things you do now with deps

and whenever we get to it, you should be able to use options to modify the -X:deps tree program too

one question I have is whether -Stree should show something like above or if it should be filtered to just the included libs like it does now

filtering can always be done with grep

$ clj -X:deps tree | grep -v "X "

I guess what I'm asking is would you prefer to usually see excluded or not?

(FWIW, I've spent a chunk of today moving us off the CLJ_CONFIG "hack" and we've decided to bite the bullet and generate our deps.edn files from a master template and a subproject template -- handling all the :override-deps directly ourselves... if the CLI / t.d.a ever supports an additional "shared project" deps.edn file, we'll switch back to that format... just got tired of dealing with tooling that doesn't honor CLJ_CONFIG and/or aliases 😞 )

Four hours of restructuring our "build" script etc and I can now run the new -X:deps tree on our subprojects 🙂

So, yeah, the new tree structure is very nice -- thanks @alexmiller -- I haven't yet seen a * case in any of our subprojects...

I changed that before I put it out, but you can check for the trailing :superseded tag

Oh, I thought * indicated the newer selected version?

not in the release, no

So you basically went the interdeps-like route?

Ah, OK. I see some :older-version tags. Haven't seen a :superseded tag yet

There were things I didn't like about how interdeps worked so I did not go back and look at it when I rebuilt our stuff today.

are you still doing a flat list of deps?

I figured you didn't want to take on some dep, that's why I added the -like suffix

if so, you'll never see it

We ended up with <subproject>/<subproject>-deps.edn for the templates and a <subproject>/<subproject>-deps.md5 file for the checksums so we can regen on-demand automatically in our build script.

or if you're always doing some stuff to use a consistent set of transitive dep versions, same thing

The checksum for each subproject includes the checksum for the master file as well, so we regenerate any <subproject>/deps.edn file if either the master template or the subproject template has changed. We have not yet tackled the issue of transitive :local/root dependencies (we probably won't, unless it actually bites us).

Just run a full pass looking for :superseded and, yup, plenty are showing up. Nice!

I'll probably roll my own solution for this too, like I did in the boot era, if this remains unsupported.

(mostly Jackson "of course")

I mean, that is actually something to maybe look at given that Jackson version changes are (often) breaking

Do you actually parse the EDN and merge yourself, or do you use simple string templating

Merge the EDN. I lift the :defaults aliases out of the master template and pull it's :override-deps out and then update the project template deps with those overrides.

yeah, easy enough.

there's like a next level of potential analysis here to gather the set of versions seen

etc etc

Yeah, it looks like we unexpectedly have some lower level pieces of Jackson that aren't being explicitly overridden...

Here's partial output:

. com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.10.2 :newer-version
      . com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.10.2 :newer-version
  X com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11 :superseded
    . com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11
    . com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11
    X com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.6 :parent-omitted
    X com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.6 :parent-omitted
  X com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11 :superseded

These are a top-level dep in a :local/root dep of this project:

. com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.8.11
    . com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.8.11

the tags you'll see at the end are really an artifact of the order they're found so that's not super relevant, but can be relevant if the set of considered versions are substantially different

you probably aren't even use those sub parts of jackson anyways

Ah, OK. So nothing to worry about here because it picks the same version anyway?

prob not