Technical Details

Architecture

Each service consists of a client script, a service script, and a Python module implementing the details of the service. The client script uploads requests and related data (e.g. an APK to sign) to an SFTP server. Then it watches the SFTP server for the response by the service. The service watches the SFTP server for incoming requests. If a request is found then the service downloads the request, processes it, and uploads the result to the SFTP server. Below, the task workflow is described in more detail.

Clients

The client scripts connect to the SFTP server and start a service-specific Job. Then they wait for the job to complete. But before they do any of this they check if the project branch the client script runs for is allowed to use the service. Typically, a service can only be used for the master branch and release branches. If the script runs for a project branch that's not cleared then the script exits with success, so that the CI system doesn't consider the CI job failed.

The Job subclass adds a new task to the task queue and uploads a Request describing the task and the files to process. When the task is completed, the Job downloads the Response and, if applicable, the processed files. Moreover, it downloads the logs captured by the service while processing the task.

Services

The server scripts connect to the SFTP server and start a service-specific Worker. Usually, the service-specific Worker subclass just defines the TaskProcessor to use.

The common Worker class watches the task queue (on the SFTP server) for incoming tasks and creates a service-specific TaskProcessor to process the task. After verifying that the task Request is legitimate (i.e. that it was submitted by a CI job running for an allowed project branch) the TaskProcessor base class calls the service-specific TaskProcessor subclass to process the Request, e.g. SignApkProcessor downloads an APK from the SFTP server, signs it and uploads the signed APK to the SFTP server. Any log output produced by the TaskProcessor is captured and also uploaded to the SFTP server.

Task Workflow

For each kind of task, e.g. signing of APKs, signing of APPXs, etc., a base folder exists on an SFTP server. Each base folder contains the following subfolders

tmp
queue
processing-<worker ID> for each worker where worker ID is a unique alpha-numeric ID
done
trash

For each task a folder containing the needed data and meta-data moves from one of those stage folders to the next one. The naming scheme for the task folders is <simplified ISO timestamp>-<client ID>, e.g. 20230516T132050-pim-itinerary_1515327. The leading timestamp allows sorting the tasks by creation time. The client ID which is built from the project path slug (CI_PROJECT_PATH_SLUG) and the job ID (CI_JOB_ID) avoids collisions.

Task Stages

Creating a new task

Clients create a new task folder in tmp and upload all necessary data to the task folder. Then they commit the task by moving it from tmp to queue.

Processing the task queue

One or more workers watch (poll) queue for new task folders. When they see a task folder, they try to take it by moving it to their processing-* folder. If they successfully took the task, they process the task, i.e. they download the data, process the data (e.g. by signing it), and upload the results. Then they set the task to done by moving it from their processing-* folder to done.

Retrieving the results

The client watches (polls) the done folder for its task folder. When the task folder is found, they download the results. Then they remove the task which currently moves the task folder from done to trash.

Cleanup

Whenever a task completes processing, the services remove old task folders from the above subfolders of the base folder. Additionally, old tasks are removed when a service is (re)started.

4 KiB Raw Permalink Blame History