Blog

I came across systemd while working with Ubuntu and other linux distributions on AWS. Its main aim is to unify service configuration and behaviour across Linux distributions and is used widely in software development. If you go through Kubernetes The Hard Way tutorial, you will find that all of the key services, e.g. Kubernetes api server, control manager, kubelet, etc. are managed using systemd.

Units

The basic object in systemd is Unit. Units can be, for example, services (.service), mount points (.mount), devices (.device) or sockets (.socket). You can see the full list of unit types using the following man command.

man systemd.unit

Custom unit files are typically put inside /etc/systemd/system folder, with the unit definition file named unit_name.service. Along with the unit file, a “drop-in” directory unit_name.service.d/ may exist. All files with the suffix “.conf” from this directory will be parsed after the unit file itself is parsed. This is useful to alter or add configuration settings for a unit, without having to modify unit files. 

A service unit file usually looks like the following:

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig="/var/lib/kubelet/bootstrap-kubeconfig" \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --image-pull-progress-deadline=2m \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --cert-dir=/var/lib/kubelet/pki/ \
  --rotate-certificates=true \
  --network-plugin=cni \
  --register-node=true \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The internal structure of unit files are organized with sections. Sections are denoted by a pair of square brackets with the section name enclosed within. Each section extends until the beginning of the subsequent section or until the end of the file. In the example, Unit, Service and Install are sections and inside sections we have key value pairs defining different directives.

For details on directives of different unit types, you can use the following commands to view more details.

man systemd.unit # View [Unit] and [Install] directives
man systemd.service # View [Service] directives

Note that unit dependencies can be defined in the Install section using WantedBy and RequiredBy directives.

Targets

Targets are basically synchronization points that the server can used to bring the server into a specific state. Service and other unit files can be tied to a target and multiple targets can be active at the same time. Target is used for grouping units and as well-known synchronization points during start-up.

systemd targets	target aliases	Description
default.target		This target is always aliased with a symbolic link to either multi-user.target or graphical.target. systemd always uses the default.target to start the system. The default.target should never be aliased to halt.target, poweroff.target, or reboot.target.
graphical.target	runlevel5.target	Multi-user.target with a GUI
	runlevel4.target	Unused. Runlevel 4 was identical to runlevel 3. This target could be created and customized to start local services without changing the default multi-user.target.
multi-user.target	runlevel3.target	All services running, but command-line interface (CLI) only
	runlevel2.target	Multi-user, without NFS, but all other non-GUI services running
rescue.target	runlevel1.target	A basic system, including mounting the filesystems with only the most basic services running and a rescue shell on the main console
emergency.target		Single-user mode—no services are running; filesystems are not mounted. This is the most basic level of operation with only an emergency shell running on the main console for the user to interact with the system.
halt.target		Halts the system without powering it down
reboot.target	runlevel6.target	Reboot
poweroff.target	runlevel0.target	Halts the system and turns the power off

Managing System Services Using Systemctl

We mainly use systemctl command to manage services on a systemd enabled server.

To view the current status of a service:

systemctl status kubelet.service

To start a service:

sudo systemctl start kubelet.service

To stop a service:

sudo systemctl stop kubelet.service

To restart a service:

sudo systemctl restart kubelet.service

To attempt to reload the service without interrupting normal functionality:

sudo systemctl reload kubelet.service

By default, most systemd unit files are not started automatically at boot. To configure this functionality, you need to enable to unit.

To enable a service:

sudo systemctl enable kubelet.service

To disable a service:

sudo systemctl disable kubelet.service

To view all active systemd units:

systemctl list-units

To include all units loaded (active + inactive):

systemctl list-units --all

View Logs Using Journalctl

A systemd component called journald collects and manages journal entries from all parts of the system. This is basically log information from applications and the kernel. We use journalctl to query logs.

To see all log entries, starting at the oldest entry:

journalctl

This will include all logs for all services therefore is not very helpful, so you will need the following commands.

To view log since the current boot:

journalctl -b

To view all logs for a service:

journalctl -u kubelet.service

To view all logs for a service during the current boot:

journalctl -b -u kubelet.service