Going from Sempahore to Jenkins on Linode to Drone CI on Inhouse computer for Ruby on Rails apps
Why did we move to own CI server?
The simple answer is fun and greed! <3
Our slowest build went from 35 minutes to 5 minutes
A bit unfair comparision of course, as the cloud computer was a 2 CPU box, while this one runs on 15 CPUs (out of 32 on our new box). Pricing compared, it's pretty close though.
We wanted more power for less. And of course, we did it for fun, it's cool running your own CI server. So cool. Ever tried that at a bar, "wanna come see my CI server?"? You probably shouldn't, unless everything else fails.
What we really wanted:
Faster build from git push to success/failure. Wait cycles leads to waste cycles. Lean: MUDA.
Semaphore is good
We were on semaphore CI and that worked great. Can't say any negative and it's probably the best solution for most :)
Most cloud CIs have a heavy start load overhead, that affects build time. To run in parallel, say 15 threads, you need to pay alot.
Building own CI Server
We embarked on a travel back in time and built our own stationary with AMD Ryzen 9 3950X 16-Core Processor.
Because parallel:spec with 32 threads sounds neat!
Why didn't we go for the AMD Ryzen Threadripper ?! 128 threads, lolpatrol! If you do, please do show and tell!
The point is to get much faster builds without overspending, so... Yeah, boring! We should get a threadripper.
TIP for running your parallel tests faster:
LIMIT the threads to 80% of available threads.
PARALLEL_TEST_PROCESSORS=9 if you're on a 6-core 12 threads laptop.
Sounds contradictory, but it's a weakest link issue, the build isn't done until the last thread is done. The computer probably has got other things to do, especially if you are running chrome, rubymine or other while waiting for your parallel tests to finish.
/end tip
Trying jenkins
First we, I, went for Jenkins. I spent days, many more hours than I'll ever admit to.
I tried first getting it running on docker, couldn't figure it out. Nobody responded satisfingly to my SO Q: https://stackoverflow.com/questions/45836492/docker-jenkins-and-rails-setup-for-running-specs-on-a-typical-rails-stack
I made it work for several rails apps. I had the most confusing, outdated libs to update github PRs and I felt helpless and stupid.
It worked though, and I was happy, eagerly waiting for the computer above to arrive. Meanwhile I had it running on a linode 2 cpu instance. It was slow, but aster than semaphore, mostly because I pre-setup database, it stored gems between builds. Overhead was similar to that of starting the build locally. That's clever, you don't need to test bundle install, yarn, database setup on every run.
I know some people are all about isolation, but that has a price. Each make their own call, I wanted this faster so it makes sense to cut the fat.
I don't want to bash on the legendary and important CI as jenkins, it's played a paramount role and I'm grateful to it's inventors and countless maintainers. That said, I wish I'd known about drone CI sooner.
Going for drone CI
drone.io is awesome!
We would probably still struggle with jenkins if my colleague Pål Sundt wouldn't be such a smartass and show he could do it better than me.
He set up a drone ci with docker, on our newly arrived stationary.
He wrote a Drone setup TODO on Rubynor blog posts:
https://rubynor.com/blog/2020/06/setting-up-drone-ci-for-rails-apps/
Next Pål set up docker images ready to build our non-docker rails apps (we got apps on heroku and some straight forward capistrano on linode)
The test runs are rawdroneously fast!
See the one project finishing in 23 secs. That includes overhead of pulling code, prepping db etc.
The setup is so simple compared to jenkins. The UI is also way simpler, less features, but great at it's primary directive, reporting on test builds.
To make it simpler for ourselves and others, we put som resources on github. The biggest issue is dockerizing builds for apps that aren't on docker, but that's not really Drone CIs problem, it' ours.
Github resources
You can find that here and I we'd be happy to put more examples there, adding contributors: