90 lines
4 KiB
Markdown
90 lines
4 KiB
Markdown
<!--
|
|
# SPDX-License-Identifier: BSD-2-Clause
|
|
# SPDX-FileCopyrightText: 2023 KDE e.V.
|
|
# SPDX-FileContributor: Ingo Klöcker <dev@ingo-kloecker.de>
|
|
-->
|
|
|
|
# Technical Details
|
|
|
|
## Architecture
|
|
|
|
Each service consists of a client script, a service script, and a Python
|
|
module implementing the details of the service. The client script uploads
|
|
requests and related data (e.g. an APK to sign) to an SFTP server. Then it
|
|
watches the SFTP server for the response by the service. The service watches
|
|
the SFTP server for incoming requests. If a request is found then the service
|
|
downloads the request, processes it, and uploads the result to the SFTP server.
|
|
Below, the task workflow is described in more detail.
|
|
|
|
### Clients
|
|
|
|
The client scripts connect to the SFTP server and start a service-specific `Job`.
|
|
Then they wait for the job to complete. But before they do any of this they
|
|
check if the project branch the client script runs for is allowed to use the
|
|
service. Typically, a service can only be used for the master branch and release
|
|
branches. If the script runs for a project branch that's not cleared then the script
|
|
exits with success, so that the CI system doesn't consider the CI job failed.
|
|
|
|
The `Job` subclass adds a new task to the task queue and uploads a `Request`
|
|
describing the task and the files to process. When the task is completed, the
|
|
`Job` downloads the `Response` and, if applicable, the processed files. Moreover,
|
|
it downloads the logs captured by the service while processing the task.
|
|
|
|
### Services
|
|
|
|
The server scripts connect to the SFTP server and start a service-specific `Worker`.
|
|
Usually, the service-specific `Worker` subclass just defines the `TaskProcessor` to use.
|
|
|
|
The common `Worker` class watches the task queue (on the SFTP server) for
|
|
incoming tasks and creates a service-specific `TaskProcessor` to process the task.
|
|
After verifying that the task `Request` is legitimate (i.e. that it was submitted
|
|
by a CI job running for an allowed project branch) the `TaskProcessor` base class
|
|
calls the service-specific `TaskProcessor` subclass to process the `Request`,
|
|
e.g. `SignApkProcessor` downloads an APK from the SFTP server, signs it and uploads the signed APK
|
|
to the SFTP server. Any log output produced by the `TaskProcessor` is captured
|
|
and also uploaded to the SFTP server.
|
|
|
|
## Task Workflow
|
|
|
|
For each kind of task, e.g. signing of APKs, signing of APPXs, etc., a base folder
|
|
exists on an SFTP server. Each base folder contains the following subfolders
|
|
|
|
- `tmp`
|
|
- `queue`
|
|
- `processing-<worker ID>` for each worker where `worker ID` is a unique alpha-numeric ID
|
|
- `done`
|
|
- `trash`
|
|
|
|
For each task a folder containing the needed data and meta-data moves from one of those
|
|
stage folders to the next one. The naming scheme for the task folders is
|
|
`<simplified ISO timestamp>-<client ID>`, e.g. 20230516T132050-pim-itinerary_1515327.
|
|
The leading timestamp allows sorting the tasks by creation time. The client ID which
|
|
is built from the project path slug (CI_PROJECT_PATH_SLUG) and the job ID (CI_JOB_ID)
|
|
avoids collisions.
|
|
|
|
### Task Stages
|
|
|
|
#### Creating a new task
|
|
|
|
Clients create a new task folder in `tmp` and upload all necessary data to the
|
|
task folder. Then they commit the task by moving it from `tmp` to `queue`.
|
|
|
|
#### Processing the task queue
|
|
|
|
One or more workers watch (poll) `queue` for new task folders. When they see a task folder,
|
|
they try to take it by moving it to their `processing-*` folder. If they successfully
|
|
took the task, they process the task, i.e. they download the data, process the
|
|
data (e.g. by signing it), and upload the results. Then they set the task to done
|
|
by moving it from their `processing-*` folder to `done`.
|
|
|
|
#### Retrieving the results
|
|
|
|
The client watches (polls) the `done` folder for its task folder. When the task
|
|
folder is found, they download the results. Then they remove the task which
|
|
currently moves the task folder from `done` to `trash`.
|
|
|
|
## Cleanup
|
|
|
|
Whenever a task completes processing, the services remove old task folders from the above
|
|
subfolders of the base folder. Additionally, old tasks are removed when a service
|
|
is (re)started.
|