Clojurians Log v2

Guys how can I use Onyx to solve the problem of executing 10 000 requests to APIs?

Do I have to implement my plugin?

or use queue?(This option will have to send the same data to the queue continuously)

@a.espolov we currently do this in two ways (there may be others) - either put a message on kafka for each request and let onyx control the concurrency / flow, or use either onyx batches ( http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-size ) or implicit batches (each segment contains data for multiple requests) and then dispatch the request batch with async i/o (we use aleph as our http client)

we commonly have 10k simultaneous api requests which get executed in onyx batches of 1k iirc

one or more server on cluster?

we are currently running 3 onyx instances

summary count cores?)

what do you mean ? how many cores across all onyx processes ?

yes

i'm not sure how useful a metric that is - we aren't cpu bound and there are other processes running on the same instances

the best answer i have off the cuff is that each onyx process is running on an aws m4.xlarge with 4 vcpus, and has 1 cpu share assigned in dc/os...

but i know from monitoring that there are between 100-200 threads (mostly in pools to service async ops) in each onyx process, so we don't have single threads monopolizing hardware threads

thanks

@a.espolov Im not sure that that's necessarily a fit for a streaming problem.

@michaeldrogalis onyx is not suitable for solving the problem periodic polling REST API?

I mean, it could. I'm just saying I'm not sure streaming is the best approach given your initial message.

ok)

@michaeldrogalis I found my issue. I needed to add [mysql/mysql-connector-java "5.1.25"] [org.postgresql/postgresql "42.1.1"] to my dependencies. Are only MySQL and PostgreSQL supported for the SQL plugin or would I be able to use something else like SQL Server?

@dbernal Theoretically anything that supports the JDBC interface ought to work

@michaeldrogalis ok perfect. Thank you!

Anytime!

@a.espolov @mccraigmccraig http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-fn-QMARK is also handy if you’re not relying on an output plugin for async batching.

oh, yeah, i should have referenced that too 😊

are there any examples available of a job that reads from a SQL table?

@dbernal The tests in onyx-sql are your best bet.

@michaeldrogalis ok cool. Thanks!