Greeting! Every xTalents out there, every single programmer, and everyone who somehow came across my blog. 😀

I want to share with you a high level overview of a project idea that I started working on a month ago.

Motivation:

I once was watching one of Kent C. Dodds great talks, and he mentioned, that while he were working in Paypal, they had an internal automated solution to boostrap and deploy a project… What? An automated CI process? We need THAT!
Couldn’t see any difference between xHub and Paypal, so I took the challenge to implement the solution! ?

I discussed the idea with the team, they approved it, so I started working on it in my free time.

Problem:

We already had a CI setup, with Jenkins and Docker, but that was a manual task, so anyone or any team who need to start a project, they have to ask for their project to be created, linked to Jenkins and deployed in our servers. That’s a time consuming process!

Solution:

The solution is pretty simple, a web interface that any user can use to bootstrap a project, after some discussions with the team, the project pipeline is as following:

1 - Create a project in Gitlab.
2 - Fork a project from an existing template.
3 - Add member as Developer to the project.
4 - Create Webhooks.
5 - Create a Jenkins Job.
6 - Build the Created Job.
7 - Deploy through Docker.
8 - Provide public URL of the project.

(Code snippet of the main function here, pseudo code)

The technical Stack of the solution:

Frontend:

• React
• Axios and Fetch to handle API calls (I’ll explain why both.)
• Context API to manage the state.

Backend:

• Nodejs
• Express
• Request (To handle API calls)
• Log4js (Minimum logging is a need!)
• Mongoose (Yep, It’s MongoDB!)

I’ll tackle few points that took me sometime to figure out. The rest and the more geeky part will be posted later on a separate Wiki post (We need to get the internal Wiki up Pokes Hakkou)

1 - Authentication:

I went so basic here, I used Password grant flow, in which a JWT Token is provided in exchange for username and password.

2 - Gitlab and Jenkins APIs:

Gitlab API is a rich API, I had minor problems working with it, so i don’t have much to say here.

Looking on Jenkins I had two ways to interact with it, either through, Jenkins CLI or Jenkins Script Console, the first choice was so limited and requires a third party jar, so I picked the second choice.

I needed a script that will do the following:

• Create a Jenkins Job
• Link it to Gitlab project
• Set the proper credentials
• Trigger a build on a push Event
• Run a Docker command after the build.

The script has to be written in Groovy! So I started my journey to find a pre-made script somewhere! ?
I couldn’t find any, but I found an interesting Jenkins Plugin called, Job DSL Plugin, that allows you to create a description of your Job in a simple structural way.

(Code snippet from the solution, here, of the DSL script)

3 - Axios and Fetch APIs:

On almost all the projects I worked on, I used to use Axios, a great package to handle your API calls, it worked on all my use cases until I stepped into this:

I had a single endpoint that I wanted to stream chunks of data (JSON), that will be sent to our React app, and update our state as the data arrives.

Since Axios is wrapped around XHR, when it comes to streaming data from the server, the entire response is buffered in memory, so you’ll always get a single response at the end (not what we want!)

I dug more on the topic, I found that XHR provides progress Event that’s fired as data arrives, but that’s mostly used when it comes to downloading files, and for that to happen, the server has to send a Content-Length header in the response.

For our use case, the length of our response is dynamic so the Content-Length is not defined in advance.

So the solution was either to use multiple endpoints or find another way to listen on a stream of data.

Fetch API!

With fetch API it’s possible to access the low-level stream as it arrives, the provided function getReader return a promise that gets resolved each time a chunk of data is received. That’s exactly what we need.

(Code snippet from the solution, here, of the DSL script)

Conclusion:

I wanted to share with you the journey of making a project, I hope you got something from it.