Clojurians Log v2

Hi! Does deps check for and download new snapshots automatically or do I / can I tell it to do so? Searching for "snapshot" at https://clojure.org/reference/deps_and_cli#_resolve_deps does not find much. 🙏

You can use -Sforce to force the cp to be recomputed

It will use the default snapshot update policy which is daily

If you want more new than that, you’ll need to remove portions of your local maven cache

I believe we have an existing ticket to allow setting the snapshot policy or more control over this

is it possible to configure update policy using global maven settings.xml?

well tools.deps doesn't use that for this so won't matter

Does deps have support for s3p:// repos, aka private s3 repositories?

A lein plugin, which our company uses is this <https://github.com/s3-wagon-private/s3-wagon-private> and I'm trying to convert a project.clj to deps.clj

just drop the p from the end of the protocol and it should work out of the box… iirc s3p is an old deprecated protocol

I will retry that, for it failed when I did it

hmmm, strange

I have this in my deps.edn...

:mvn/repos {"foo-releases" {:url "<s3://maven.foobar.com/releases>"}
             "foo-snapshots" {:url "<s3://maven.foobar.com/snapshots>"}}}

but the resolution does this:

Downloading: foo/components/0.2.0/components-0.2.0.pom from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.pom from foo-snapshots
Downloading: org/clojure/clojure/maven-metadata.xml from foo-snapshots
Downloading: org/clojure/clojure/maven-metadata.xml from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.jar from foo-releases
Downloading: foo/components/0.2.0/components-0.2.0.jar from foo-snapshots
Error building classpath. Could not find artifact foo:components:jar:0.2.0 in central (<https://repo1.maven.org/maven2/>)

Not sure what is going on at all, why is it calling out to maven central (no artifact has been downloaded)

I’m pretty sure that’s just because it looks for the deps on all repos, which is why you’re seeing that

Okay, a fallback. Fairy enough. However, it appears not to be able to download the artifact on s3.

it's not deprecated, it's just specific to that leiningen plugin. that's not a "real" thing

correct on the error message - it cycles through all repos and will report only the last error

Ah, I think I've also discovered why it didn't download

double checking

is that a private s3 repo or public?

private

and you provided auth ?

that's what I'm double checking atm 🙂

verifying my setup

https://clojure.org/reference/deps_and_cli#_maven_s3_repos

also important is that the AWS creds you use have a policy that allows s3 ops

yup, all that works with lein, I'm doing a conversion to deps

but your log snippet above shows it successfully downloading stuff from the repo

No, it didn't

it created the directories, but didn't download anything.

so that message is misleading

(I wouldn't report it not downloading anything, without first checking if the directories actually contained stuff)

do you have metadata files in there or are they literally empty?

literally empty

metadata would have indicated that it yes, downloaded something (then my question would have been different, i.e., why metadata but no jar...)

can you try again with -Sforce ?

That's what I have been using

ok

So, I got it to download now, like this:

where are you putting AWS creds?

in my .m2/settings.xml I had the the username/password set to my access key and secret access key (when I first attempted to verify that the conversion from project.clj to deps.clj was okay)

(following the instructions on the <https://clojure.org/reference/deps_and_cli#_maven_s3_repos> link)

that didn't work (see eyes passim)

but then I removed the u/p combo

and used this way

aws-vault exec foo -- clj -Sforce

what's aws-vault?

then that worked

<https://github.com/99designs/aws-vault>

it does stuff in the background to setup credentials and environment variables

(but <shrug> I'm quite new to using it too, it's used at a company I've just joined)

I see

temporary ambient creds

right, it has things like this

aws-vault list

aws-vault list
Profile                  Credentials              Sessions                      
=======                  ===========              ========                      
default                  -                        -                             
dharrigan                -                        -                             
foo                       foo                       sts.GetSessionToken:5h54m59s  
foo:staging               -                        -                             

may be that whatever your ambient aws creds were were getting used over the user/pw creds

I think it's the session credentials that should replace what was in the .m2/settings.xml, (I've removed the access key/secret key combo from the .m2/settings.xml)

If I do this:

❯ aws-vault exec foo -- env | rg AWS  
_=/usr/bin/aws-vault
AWS_VAULT=foo
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ACCESS_KEY_ID=wibble
AWS_SECRET_ACCESS_KEY=wobble
AWS_SESSION_TOKEN=wibble-session
AWS_SECURITY_TOKEN=wobble-sesion
AWS_SESSION_EXPIRATION=2021-06-25T20:08:23Z

I can see that a lot of env variable are set, which I suppose the S3Transporter uses to successfully download the jar from.

Should this be documented for those who come after me? A company that has a security policy like this, that requires that session tokens are gained and used, will affect anyone trying to download from s3 without this knowledge.

should what be documented?

s3 ambient creds are discussed in the docs

The instructions on the <https://clojure.org/reference/deps_and_cli#_maven_s3_repos> for accessing private s3 repos do not take into account temporary session credentials

I don't think this For authenticated repos, AWS credentials can be set in the ~/.m2/settings.xml on a per-server basis or will be loaded ambiently from the AWS credential chain (env vars, etc). is quite sufficient detail

they certainly hint at it, but don't expound upon it

I'm not sure what's special about the temporariness of the credentials

this is the relevant part:

AWS S3 credentials can be set in the environment using one of these mechanisms:

    Set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

    Create a default profile in the AWS credentials file ~/.aws/credentials (older ~/.aws/config also supported).

    Create a named profile in the AWS credentials file and set the environment variable AWS_PROFILE with its name.

    Amazon ECS container and instance profile credentials should also work, but have not been tested.

For more information, most of the advice in this AWS document describes how credentials are located. Note however that the Java system properties options will NOT work with the command line tools (but would work if using the tools.deps.alpha library directly).

so, I think it does expound on it?

It's missing information on the session tokens, which are used in tandem with the access key and secret access key - but only under specific circumstances, i.e., when there is an AssumeRole going on.

(otherwise,the access key and secret access key, set in the .m2/settings.xml under username/password would have worked out-of-the-box)

I'm highlighting a gap in the explanation that I came across. With the thought of being helpful to others in the future, who may encounter this too, I thought it would be beneficial to document it somewhere.

so you're talking about user/pw being insufficient when using session tokens

I think so yes, it wasn't sufficent for me when I had them in the .m2/settings.xml. However, let me validate that again by doing another test. I'll report back 🙂