Clojurians Log v2

Hello, @wilkerlucio — after listening to all your interviews with @jacek.schae, I was blown away, and watched all the videos you linked to. Congrats on continuing to build out such a remarkable and novel way managing data! After seeing the magic that’s possible, I suddenly find it quite distasteful to have to write API and database queries over and over again, and often, once on the client side, and another time on the server side! I may have been mildly annoyed about doing that before, but after seeing the SpaceX and YouTube demo that @pithyless did, I now know that it doesn’t have to be that way — and suddenly, writing the same queries in two portions of code seems intolerable! 😆 😆 😆 I’ve spent about 6 hours trying to write a resolver against the Vimeo — it’s been kind of tough sledding, because it’s taken me awhile to comprehend sufficiently how all the pieces get created. I found myself watching the “Om Next” video from @wilkerlucio several times, pausing many times to watch how he built the Spotify example. I just published my work in progress here — my goal is to document my experiences trying to write a Vimeo adapter, point out where I had sticking points, and propose some doc changes. I’ll write that up when I’m a little further down, but wanted to share progress so far, just in case it’s useful (or amusing. 🙂 I’ve written the easiest resolver (“video-by-id”), and I’m on the third attempt to get “albums-by-user-id” written — I got hung up on how to deal with the collection inside of the return value… https://github.com/realgenekim/pathom-demo/tree/gene-vimeo-hacking-away Happy to hear any critiques or recommendations on the work I’ve done so far — here’s to hoping I can get “albums-by-user-id” working in this work session tonight! Thank you, @wilkerlucio and @pithyless!

(Sorry, meant to post more, but need to run — but wanted to post something now, as opposed to more later. 🙂

Okay, for the life of me, I can’t seem to get the Vimeo equivalent to the spotify.top-tracks to work, as was shown in this repo: https://github.com/wilkerlucio/spotify-pathom/blob/master/src/spotify_pathom/parser.clj#L64 I manually assoc in a vimeo.album-list.data -> vimeo.album/user-id, creating an alias with that and vimeo.user/id. And I just can’t figure out how to write the equivalent EQL of the spotify-top-tracks. How do I get all the album URLs for a given user? Blindly copying the spotify query, I feel like it should be something like in the screenshot below from the video. Any hints much appreciated!!! 🙏🙏

Here’s the relevant part of the 2017 “beyond Om Next video: https://youtu.be/60i9uStI9As?t=1216

hello @genekim! happy to see your excitement 🙂

@genekim not sure if I understand your question, can you send the example query you are trying to run? what you expect, and what is happening?

@wilkerlucio there is some function to remove the "cache" from env? I can do a "select-keys" and get just the index keys, but it will remove my custom env vars. Something like:

(pco/defresolver foo [env _]
  (let [... (p.eql/process (pci/without-cache env)
                           [::my-key])]
    ...))

(dissoc env
  :com.wsscode.pathom3.connect.runner/resolver-cache*
  :com.wsscode.pathom3.connect.planner/plan-cache*)

this will create new sub caches on run, if you really want to disable cache entirely, then:

(assoc env
  :com.wsscode.pathom3.connect.runner/resolver-cache* nil
  :com.wsscode.pathom3.connect.planner/plan-cache* nil)

I suggets keeping plan-cache*, but I mentioned here just for completeness

@wilkerlucio Thx for the focusing question. My query that isn’t working looks like

[{[:vimeo.user/id 118038002]
  [* :vimeo.album-list/data
   [[:vimeo.album/uri]]]}]

{[:vimeo.user/id 118038002
  [:vimeo.album/uri "uri1"]
  [:vimeo.album/uri "uri1"]])
 

{[:vimeo.user/id 118038002]
 {:vimeo.album-list.paging/previous nil,
  :vimeo.album-list.paging/next nil,
  :vimeo.album-list.paging/last "/118038002/albums/?page=1",
  :vimeo.album-list/per-page 25,
  :vimeo.album-list.paging/first "/118038002/albums/?page=1",
  :vimeo.album-list/id 118038002,
  :vimeo.album-list/page 1,
  :vimeo.album-list/total 8,
  :vimeo.album-list/data
  [{:vimeo.album/uri "/users/118038002/albums/7670034",
    :vimeo.album/autoplay false,
    :vimeo.album.user/link "<https://vimeo.com/user118038002>",
    :vimeo.album.metadata.interactions.add-logos/uri
    "/users/118038002/albums/7670034/logos",
    :vimeo.album.user.preferences.videos.privacy/view "disable",
    :vimeo.album.user.metadata.interactions.block/uri
    "/me/block/118038002",
    :vimeo.album.user.metadata.connections.channels/total 0,
    :vimeo.album.user.metadata.interactions.follow/added false,
    :vimeo.album.user.metadata.connections.teams/options ["GET"],
    :vimeo.album.user.metadata.connections.folders-root/uri
    "/users/118038002/folders/root",
    :vimeo.album.user.pictures/resource-key
    "9b6cd50b7543a03d77365da646638c8e06f4c7ea",
    :vimeo.album.user.metadata.connections.followers/options ["GET"],
    :vimeo.album.metadata.interactions.add-live-events/options
    ["POST"],
    :vimeo.album.user.metadata.connections.folders/uri
    "/users/118038002/folders",
    :vimeo.album.metadata.connections.videos/uri
    "/albums/7670034/videos",
    :vimeo.album/brand-color nil,
    :vimeo.album.user.metadata.connections.videos/options ["GET"],
    :vimeo.album.privacy/view "anybody",
    :vimeo.album.user.metadata.connections.appearances/total 0,
    :vimeo.album/seo-keywords [],
    :vimeo.album.user.metadata.connections.portfolios/uri
    "/users/118038002/portfolios",
    :vimeo.album/name "Shaaron",
    :vimeo.album.pictures/sizes
    [{:width 100,
      :height 75,
...

…I can’t quite figure out what the EQL is for doing joins in collections… and I keep wondering whether I wrote the resolver incorrectly. Thx!

@genekim I think you're missing a join:

{[:vimeo.user/id 118038002]
 [{:vimeo.album-list/data [:vimeo.album/uri]}]}

Joins use the same syntax for to-one and to-many relationships; the way you tell them apart is by returning {:join-key {:a :b}} or {:join-key [{:a :b}]

Holy cow — that’s cool! I’m getting this error in the parser — does this mean I’m missing a resolver? (PS: thanks for your talk on SpaceX/YouTube. It was what made me finally feel like I should jump in and try to make this work!!!)

^ in both cases, the EQL would be the same: [{:join-key [:a]}]

It looks like the resolver indices are not set up correctly and can't resolve this relation

@genekim does that :vimeo.album-list/data autocomplete?

Got it…. and this query suggests that, as well… Any chance you can see what I did wrong in either my attrs or resolvers? Resolver.cljs: https://github.com/realgenekim/pathom-demo/blob/gene-vimeo-hacking-away/src/demo/connect/vimeo/resolvers.cljs Attrs.cljs: https://github.com/realgenekim/pathom-demo/blob/gene-vimeo-hacking-away/src/demo/connect/vimeo/attrs.cljs#L283

@pithyless Something is definitely off — what does it mean when something displayed in red vs. brown in the parser window?

There's a lot going on here, try to simplify it for now:

(def albums-by-user-id
  (pc/defresolver albums-by-user-id [env input]
    {::pc/input  #{:vimeo.user/id}
     ::pc/output [{:vimeo.album-list/data [:vimeo.album/user-id]}]
     {:vimeo.album-list/data [{:vimeo.album/user-id "fake-user-id"}]}}))

Does that work with this query?

{[:vimeo.user/id 118038002]
 [{:vimeo.album-list/data [:vimeo.album/uri]}]}

OMG! It works! 🎉🎉🙏🙏 THANK YOU! Holy cow! That means code is okay, right? This is fantastic!

cool 😎

Gotta step away from keyboard for an hour, but that query worked without the code change that you proposed — whew! …and can you explain the EQL syntax you used? In desperation, I thought I had tried every permutation of “[” and “{”, and still somehow managed to miss this. Is there a section in the docs that explains this type of join against a collection?

When debugging resolvers, two common things to look for: 1. is the autocomplete working and are the indices aware of the relation I'm trying to join 2. are the resolvers returning the right shape? (e.g. accidentally returning [{:name "a"}] instead of {:join-key [{:name "a"]} if the resolver output is supposed to be [{:join-key [:name]}]

Have you seen the EQL docs? https://edn-query-language.org/eql/1.0.0/specification.html#_joins

@genekim here's a step-by-step process how to think about the query you just wrote:

Start with a vector of attributes:

[SOME_ATTRS]

One of the attrs should be a join, so we need a map:

[{SOME_JOIN_KEY [SOME_JOIN_ATTRS]} ...POSSIBLY_MORE_ATTRS...]

In this case, we only care about the one attribute:

[{SOME_JOIN_KEY [SOME_JOIN_ATTRS]}]

What's the join key? An ident in this case:

[{[:vimeo.user/id 118038002] [SOME_JOIN_ATTRS]}]

What are the attributes we're interested in? Well, one of them is a join:

[{[:vimeo.user/id 118038002] [{NEW_JOIN_KEY [NEW_JOIN_ATTRS]} MORE_ATTRS]}]

Actually, we only care about that one join attribute:

[{[:vimeo.user/id 118038002] [{NEW_JOIN_KEY [NEW_JOIN_ATTRS]}]}]

What's the join key?

[{[:vimeo.user/id 118038002] [{:vimeo.album-list/data [NEW_JOIN_ATTRS]}]}]

What are we trying to pull from that relationship?

[{[:vimeo.user/id 118038002] [{:vimeo.album-list/data [:vimeo.album/uri]}]}]

And in case you missed the other thread response: Joins use the same syntax for to-one and to-many relationships; the way you tell them apart is by returning `{:join-key {:a :b}}` or `{:join-key [{:a :b}]` In both cases, the EQL (and the resolver output) would be the same: `[{:join-key [:a]}]`

Wow! Thank you @pithyless! That is so helpful!!! I’ll be studying this for days to come, I’m sure. Much appreciated, and I’ll endeavor to write up my experiences when I get a little further, and maybe some suggestions on docs to make these more evident.

Thanks again for your help, @pithyless! Does anyone have any advice on how to handle paginated response? As shown below, vimeo.album-list includes some information of how many pages are in the total response, URL of the next page, etc… How do people handle this? Do you pass in a page-number to vimeo.album-list? Do you load all the pages by default? (<-- I imagine this would be problematic for many scenarios. I’m imagining an API into Google Photos, which would load 100K entries upon every query. 🙂 Are there any examples you’d point to of how others have handled pagination? Thanks again!

{[:vimeo.user/id 118038002]
 {:vimeo.album-list.paging/previous nil,
  :vimeo.album-list.paging/next nil,
  :vimeo.album-list.paging/last "/118038002/albums/?page=1",
  :vimeo.album-list/per-page 25,
  :vimeo.album-list.paging/first "/118038002/albums/?page=1",
  :vimeo.album-list/id 118038002,
  :vimeo.album-list/page 1,
  :vimeo.album-list/total 8,
  :vimeo.album-list/data [{} {} ...]}

…ah, it looks like the preferred method is passing in a cursor-like thing as a parameter to vimeo.album-list… I’m assuming this is still the case? https://clojurians-log.clojureverse.org/pathom/2018-12-10

@genekim I would very much appreciate more ideas to improve the experience on early users 🙂 both for the current pathom docs, and also for pathom 3, had you checked the docs over pathom 3? altough the usage has some differences, the fundamentals are the same, and the docs on pathom 3 reflect the most mature way I can explain those concepts

Yep, I’ve read the current and Pathom 3 docs. FWIW, I think there’s one piece of “getting started” docs missing, which would include the EQL docs that @pithyless just told me about today… …as well as more about the procedures of how to create the resolvers, the attributes, etc. I plan on drafting a HOWTO for my own purposes in that repo, so I can finish building out the Vimeo API. I’ll hopefully have something to share with you in about a week, which I”m hoping would be helpful to anyone who wants to get started. It’s super awesome to see my query work finally, and eager to now start pulling individual albums and videos next!

…I plan on documenting what it is I’m studying and copying when I watch certain portions of your “Om Next” video over and over again.