supraj.dev

Interview questions & answers

Linux

Processes, signals, networking, and troubleshooting.

  • 148 questions
  • 5 easy
  • 63 medium
  • 80 hard
  • 1 related article

148 questions

Question 1Easy

What is the difference between a hard link and a symbolic link?

#
Show answer

A hard link is another directory entry pointing to the same inode — the file exists as long as any hard link remains, and all links are equal. A symlink is a small file that stores a path to another file; it can cross filesystems and link directories, but breaks if the target is moved or deleted. Hard links can't cross filesystems or point to directories.

Question 2Easy

How do you find which process is using a port?

#
Show answer

Use ss -ltnp 'sport = :8080' (or lsof -i :8080, or older netstat -ltnp) to list the listening socket and its PID/program. Then inspect or kill it with ps -p <pid> / kill. ss is the modern, faster replacement for netstat.

Question 3Medium

A server is slow. Walk through how you triage it.

#
Show answer

Start broad with top/htop and uptime (load average vs core count). Then split by resource: CPU (top, pidstat), memory (free -h, check swap and OOM in dmesg), disk I/O (iostat -x, iotop), and network (ss, iftop). vmstat 1 gives a quick overall view. The goal is to localize the bottleneck to CPU, memory, I/O, or network before drilling into the offending process or query.

Question 4Medium

What do the permission bits 755 and the SUID bit mean?

#
Show answer

755 = owner rwx (7), group r-x (5), others r-x (5) — common for executables and directories. SUID (e.g. 4755, the leading 4) makes an executable run with the file owner's privileges rather than the caller's — that's how passwd edits /etc/shadow as root. SUID on binaries is a security-sensitive setting and a common privilege-escalation vector.

Question 5Hard

What is the difference between SIGTERM, SIGKILL, and SIGHUP?

#
Show answer

SIGTERM (15) politely asks a process to terminate — it can catch it, clean up, and exit; this is what you should use first and what kill sends by default. SIGKILL (9) cannot be caught or ignored; the kernel kills the process immediately with no cleanup — last resort. SIGHUP originally meant terminal hangup but by convention tells daemons to reload their config without restarting. Graceful shutdown handlers in apps/containers hook SIGTERM.

Question 6Hard

A process inside a Docker container (which uses cgroups v2) reports memory usage of 4GB via free -h, but the host shows the container's cgroup memory limit is 2GB. What is happening, and how does cgroup v2 memory accounting differ from the kernel's view of the process?

#
Show answer

free -h reads from /proc/meminfo, which shows the host's global memory statistics, not the container's cgroup limits. In cgroup v2, the container's memory limit is enforced at the cgroup level, not via proc. The correct way to see container memory usage is cat /sys/fs/cgroup/memory.current and cat /sys/fs/cgroup/memory.max. The 4GB shown by free is the host's total memory. cgroup v2 memory accounting includes: memory.current (total page cache + RSS + swap), memory.swap.current (swap usage), and memory.stat for detailed breakdown (anon, file, kernel, slab). Unlike cgroup v1, v2 unifies memory and memsw accounting — the memory.max limit applies to memory+swap combined. The kernel enforces OOM via memory.events — when max is exceeded, oom_kill increments. If free shows 4GB, the process is seeing host memory. For accurate container-level memory, always read the cgroup files. Docker's docker stats reads these cgroup values correctly.

Question 7Hard

A systemd service fails to start with "Unit exited due to start timeout". The ExecStart command runs a Python script that sometimes takes 5 minutes. How do you reconfigure systemd without changing the script, and what's the default timeout?

#
Show answer

The default TimeoutStartSec is 90 seconds (systemd v240+). When the script exceeds this, systemd sends SIGTERM (configurable via KillSignal), then SIGKILL after TimeoutStopSec. To fix: override the timeout in a drop-in via systemctl edit myservice.service and add [Service] TimeoutStartSec=300 (or infinity). However, if the script hangs indefinitely, this masks the problem. Better: use Type=notify with sd_notify("READY=1") in the script, so systemd knows the service is alive without waiting for the full timeout. If the script is not modifiable, use Type=simple with ExecStartPost that polls for readiness — or use Restart=on-failure with RestartSec=30 to handle transient delays. Also consider TimeoutStartSec behavior differs between Type=simple (systemd waits for the execve only) and Type=oneshot or Type=notify (waits for process exit or notification). For long initialization, Type=idle delays start until boot completes.

Question 8Medium

A developer runs strace -p <PID> on a stuck process and sees futex(0x7f..., FUTEX_WAIT_PRIVATE, 2, NULL) repeating. What does this mean, and how do you identify which mutex is contended?

#
Show answer

futex(FUTEX_WAIT_PRIVATE) means the thread is blocked waiting on a userspace mutex. The futex syscall is the kernel-level wait operation — when FUTEX_WAIT_PRIVATE returns EWOULDBLOCK, the thread re-checks the userspace lock word; if still contended, it sleeps. The address 0x7f... is the userspace address of the futex word (part of the pthread_mutex_t or pthread_cond_t). To identify the contended mutex: (1) Check /proc/<PID>/maps to find which shared library or heap region maps that address. (2) Run gdb -p <PID>, then info threads, thread apply all bt to see which thread holds the lock. (3) Use futex traces from all threads — the lock owner is not in FUTEX_WAIT but in FUTEX_WAKE or running code. (4) Enable lockdep with echo 1 > /sys/kernel/debug/tracing/events/lock/lock_acquire/enable. (5) Use perf lock record -p <PID> then perf lock report to show contended locks by symbol. For Go programs, use GOTRACEBACK=all or the runtime/trace output instead.

Question 9Hard

Explain how Linux namespaces interact with cgroups to provide container isolation. Specifically, how does a user namespace map root (UID 0) inside the container to an unprivileged UID on the host, and what are the security implications?

#
Show answer

User namespaces map a range of UIDs inside the namespace to a different range on the host via /proc/<PID>/uid_map. For example, 0 100000 65536 maps container UID 0 to host UID 100000. This means the container's root has no special privileges on the host — kernel capabilities are scoped to the namespace. When a process in the container tries to mknod or setcap, the kernel checks the mapped host UID, not the namespace UID. Without user namespaces, containers share the host's UID space — if allow userns=0 isn't enabled, an escape from a root-in-container process becomes real root on the host. However, user namespaces add complexity: (1) Mounting filesystems requires --privileged or CAP_SYS_ADMIN in the initial namespace. (2) The subuid and subgid files (in /etc/) define the mapping ranges used by tools like Podman. (3) Not all kernel interfaces are namespace-aware — for example, /proc/sysrq-trigger is global. Docker by default blocks user namespace remapping for networking (since netdev ownership is global), so rootless containers in Docker require extra networking setup (slirp4netns).

Question 10Hard

You run a production database on ext4. Write-heavy workloads cause jbd2 process to consume high CPU and I/O wait spikes. Diagnose the root cause and explain if XFS would behave differently.

#
Show answer

jbd2 is the ext4 journaling daemon. High CPU/IO means the journal is a bottleneck — every metadata write goes to the journal first (write-ahead logging). ext4 defaults to data=ordered mode: data blocks are written before metadata is journaled. Under high write concurrency, the journal commits become serialized, and jbd2 flushes the journal to disk at each commit (default 5-second commit interval, or sooner if the journal fills). XFS uses a different design: it allocates log space in a circular buffer, groups multiple metadata updates into a single transaction, and uses delayed logging — dirty metadata is accumulated in memory and committed asynchronously. XFS typically handles concurrent metadata writes better because it batches more aggressively and its log writes use asynchronous I/O when the storage supports it. However, both filesystems will show jbd-like activity under extreme write pressure. Mitigations for ext4: increase the journal size (128MB default), place journal on a separate SSD/NVMe device, use barrier=0 if you accept crash-risk, or switch to data=writeback (metadata-only journaling, risk of data corruption on crash).

Question 11Medium

A server runs with swap enabled (swappiness = 60). The application starts showing high latency even though free -h shows 70% RAM free. vmstat shows si and so are non-zero. What's happening, and how do you fix it?

#
Show answer

This is premature swapping. The kernel's page reclaim logic may swap out inactive anonymous pages even when plenty of file-backed page cache is available. Swappiness (default 60) controls the balance between reclaiming file-backed pages vs anonymous pages — at 60, the kernel is slightly biased to reclaim file-backed pages. If the working set includes many mapped pages (e.g., Java heap, mmap'd files), the kernel may swap them to avoid evicting cached files. The fix: sudo sysctl vm.swappiness=1 (or 0 on kernels < 5.8+ where 0 disables swap entirely). Also check vm.vfs_cache_pressure — lowering it retains dentry/inode caches. Additional diagnostics: sar -S to see swap rates, cat /proc/meminfo | grep -i swap to find what's swapped (use smem or swap from procps-ng). If the application is Java, verify the JVM heap fits in RAM and consider -XX:+UseTransparentHugePages to reduce TLB misses. For databases, disable swap entirely (swapoff -a) or set swappiness=1 and configure vm.overcommit_memory=2 to prevent memory-overcommit-based swapping.

Question 12Medium

What is the practical difference between SELinux's targeted and strict policies, and how would you troubleshoot a process that gets "AVC denied" even though the file permissions (rwx) are correct?

#
Show answer

targeted applies SELinux policy only to specific "targeted" daemons (httpd, sshd, named) while leaving other processes unconfined. strict applies policy to all processes, including user sessions — rarely used in production because it breaks almost everything unless the policy is custom-tailored. An "AVC denied" with correct rwx means the file's SELinux context (user:role:type:level) does not match the process's domain. For example, httpd runs in httpd_t domain, but the file is labeled user_home_t (default for files in /home/). Fix: chcon -t httpd_sys_content_t /path/to/file or semanage fcontext -a -t httpd_sys_content_t /path(/.*)? then restorecon -Rv /path/. To troubleshoot: run ausearch -m avc -ts recent | audit2why to decode the denial. audit2allow generates a local policy module to permit the access, but in production, you should instead set the correct context. For debugging temporarily: setenforce 0 (permissive mode) — but never leave it disabled in production.

Question 13Hard

A systemd timer triggers a service every 5 minutes, but sometimes the service takes 7 minutes. The timer starts a new instance anyway, causing overlapping runs. How do you prevent overlap — using flock in the script vs using systemd's unit options?

#
Show answer

In the service unit: set [Service] Type=simple with no concurrency control — systemd will start a new instance whenever the timer fires. To prevent overlap, use [Unit] Conflicts=myservice.service combined with [Service] RemainAfterExit=yes — but this is convoluted. The idiomatic systemd solution: in the timer unit, set Unit=myservice.service without OnCalendar only — that doesn't prevent overlap. Instead, configure the service to be Type=oneshot and add [Service] StartLimitIntervalSec=0 with Restart=no, and in the timer, set OnCalendar=*:0/5 with AccuracySec=1s — but this still allows overlap. The correct approach: set [Unit] in the service to StopWhenUnneeded=no and use [Service] ExecStartPre=/usr/bin/flock -n /var/lock/myservice.lock. Better: add [Unit] ConditionPathExists=!/var/lock/myservice.lock and have the script create the lock. The cleanest systemd-native approach: use a [Unit] with BindsTo=myservice.service and After=myservice.service but that's tricky with timers. The practical answer: use flock -n /tmp/myservice.lock /path/to/script in ExecStart — the -n exits immediately if lock is held, and FailureAction= in the timer can handle errors.

Question 14Medium

A process is using 100% CPU but you cannot tell from top which code path is hot. What tools beyond top do you use to identify the exact function in a compiled binary?

#
Show answer

top shows aggregate CPU but not per-function breakdown. Use perf top -p <PID> to see the hottest functions in real-time, with symbol names. For JIT-compiled code (Node, JVM, .NET), use perf top -p <PID> -k 0 (to skip kernel symbols) with -g for call-graph. If symbols are missing (stripped binary), install debug symbols or use perf top -f for raw instruction-pointer sampling. Other tools: flamegraph via perf record -F 99 -g -p <PID> -- sleep 30 && perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg. For Python: py-spy or cProfile. For Java: async-profiler. For short bursts, use perf stat -p <PID> -e cycles,instructions,cache-misses,branch-misses -- sleep 10 to see microarchitectural bottlenecks, then drill with perf record and perf annotate. If the binary is compiled with -fno-omit-frame-pointer, call graphs are accurate; without it, perf uses DWARF unwinding (--call-graph dwarf).

Question 15Hard

You are on a system without cron installed, and you cannot install packages. Only systemd is available. Write a persistence mechanism that runs a cleanup script every 6 hours without a timer unit file?

#
Show answer

You can create a transient systemd timer using systemd-run. The command: systemd-run --user --on-calendar=daily --unit=cleanup /path/to/script.sh — but for 6-hour intervals: systemd-run --on-active=6h --unit=cleanup6h /path/to/script.sh. However, --on-calendar requires proper calendar syntax. For a repeating 6-hour schedule: systemd-run --on-calendar="*:0/360" --unit=cleanup /path/to/script.sh. If you cannot use --on-calendar, use systemd-run --unit=cleanup.service --timer-property=OnUnitActiveSec=6h /path/to/script.sh. This creates a transient timer and service unit in /run/systemd/transient/ that persists until next reboot. For reboot persistence without package installs: write a service unit to /etc/systemd/system/ directly (no systemctl daemon-reload needed if you use systemctl link). If even file write is restricted, use systemd-run --user --scope to create a scope, but it won't survive reboot. The --user instance requires lingering enabled (loginctl enable-linger). The --on-active creates a monotonic timer that runs once — for repeating, use OnUnitActiveSec in the timer properties.

Question 16Medium

An NFS mount on your Linux server causes ls -la to hang for 30+ seconds when the NFS server is slow. How do you diagnose which mount is hanging and fix it without rebooting?

#
Show answer

Diagnose: (1) Run mount | grep nfs to list all NFS mounts. (2) Run cat /proc/mounts | grep nfs and check the soft vs hard option — hard mounts block I/O indefinitely if the server is unreachable. (3) Use strace -p <PID_of_ls> to see if it's stuck on stat or getdents on an NFS path. (4) Run nfsstat -m for per-mount statistics. (5) Quick fix for a hanging mount: umount -f -l /path/to/mount (force lazy unmount) — the filesystem is detached immediately, processes get I/O errors and resume. (6) For a soft recovery: mount -o remount,soft,timeo=100,retrans=3,retry=0 /path — soft causes operations to timeout and return errors instead of hanging; timeo (tenths of seconds) sets the retransmission timeout. (7) Long-term: use soft,intr,timeo=600 for NFSv3 or soft,timeo=600 for NFSv4; monitor NFS server health and network latency; use automount (autofs) to unmount idle NFS shares automatically.

Question 17Hard

Your production server disk is at 100% according to df -h but du -sh / shows only 60% usage. What causes this and how do you recover?

#
Show answer

Discrepancy means a process holds open file descriptors to deleted files. When a file is deleted while open, du no longer counts it (no directory entry), but df still includes it because the inode is referenced. Find the offenders: lsof | grep "(deleted)" | awk '{print $2, $NF}' to list PID and file path of deleted files. Then check ls -la /proc/<PID>/fd/ | grep deleted for exact sizes. Recovery options: (1) Kill the process holding the file — the kernel releases the space: kill <PID>. (2) If the process cannot be restarted, truncate the file: : > /proc/<PID>/fd/<FD_NUM> — this empties the file content in-place, releasing disk space without interrupting the process. (3) For log files that grow continuously, configure logrotate with copytruncate (copies log then truncates) instead of create (creates new inode) to avoid this scenario. Preventive monitoring: alert on disk_usage > 80% and track df -i for inode exhaustion separately.

Question 18Hard

A process uses 99% CPU but the OOM killer keeps killing other processes instead. Explain the OOM scoring algorithm and how to target the correct process.

#
Show answer

The OOM killer uses oom_score = total_rss + total_swap + (total_pages * page_table_factor) / oom_score_adj. The key factor: oom_score_adj (range -1000 to 1000). By default, all processes have oom_score_adj=0. A CPU-intensive process that uses little memory (most CPU time is spent in tight loops on cached/pinned data) has a low oom_score because RSS and swap are low. The OOM killer targets the process with the highest oom_score, which is typically a memory-intensive process (database, cache) that happens to be using lots of RAM. To protect memory-heavy but critical processes, set echo -500 > /proc/<critical_pid>/oom_score_adj (less likely killed). To target the CPU-hungry process: echo 500 > /proc/<cpu_hog_pid>/oom_score_adj which increases its score. But the best fix is addressing the actual problem — either reduce memory pressure or limit the CPU hog. Use systemd-run --scope -p MemoryMax=<limit> -p CPUQuota=<limit> <command> to constrain the process. For production: never rely on OOM killer — set appropriate cgroup limits (memory.max in cgroup v2) per service so the kernel OOMs only within the cgroup, not globally.

Question 19Hard

Linux incident: A Linux server shows 100% CPU but low application throughput. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: top, mpstat, pidstat, perf top, run queue length, and application profiling. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Question 20Hard

Linux architecture scenario: You need to separate CPU-bound work from lock contention and IO wait. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Question 21Medium

Linux security scenario: A quick restart hides the evidence needed for root cause. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Question 22Hard

Linux release scenario: You are changing CPU limits and process placement. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Question 23Medium

Linux reliability/cost scenario: Peak-hour CPU alarms are firing every day. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Question 24Hard

Linux incident: The kernel OOM killer terminates critical processes at random. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: dmesg, journalctl, /proc/meminfo, smem, ps aux --sort -rss, and cgroup memory stats. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Question 25Hard

Linux architecture scenario: You need predictable memory behavior and early warning. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Question 26Medium

Linux security scenario: Swap hides memory leaks until latency collapses. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Question 27Hard

Linux release scenario: You are introducing systemd memory controls. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Question 28Medium

Linux reliability/cost scenario: Memory upgrades are increasing cost without fixing leaks. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Question 29Hard

Linux incident: The root filesystem fills up and services fail to write logs or temp files. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: df -h, du -xhd1, lsof +L1, journal size, inode usage, and logrotate status. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Question 30Hard

Linux architecture scenario: You need disk isolation for logs, data, and system partitions. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Question 31Medium

Linux security scenario: Deleting files that are still held open does not free space. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Question 32Hard

Linux release scenario: You are moving logs and data to separate mount points. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Question 33Medium

Linux reliability/cost scenario: Emergency disk cleanup is becoming routine. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Question 34Hard

Linux incident: Only one service path has high latency from a Linux host. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: curl -w, dig, ss -ti, mtr, tcpdump, MTU checks, and conntrack stats. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Question 35Hard

Linux architecture scenario: You need to isolate DNS, TCP, TLS, routing, and application latency. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Question 36Medium

Linux security scenario: Packet captures may expose sensitive payloads. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Question 37Hard

Linux release scenario: You are changing MTU or routing in production. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Question 38Medium

Linux reliability/cost scenario: Retries are amplifying downstream load. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Question 39Hard

Linux incident: A systemd service repeatedly restarts but manual command execution works. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: systemctl status, journalctl -u, systemd-analyze verify, unit environment, and exit codes. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Question 40Hard

Linux architecture scenario: You need reliable service units with correct environment and dependencies. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Question 41Medium

Linux security scenario: Running services as root masks permission issues. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Question 42Hard

Linux release scenario: You are adding hardening options to systemd units. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Question 43Medium

Linux reliability/cost scenario: Restart loops page on-call with no useful context. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Question 44Hard

Linux incident: A service fails with too many open files during traffic spikes. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: lsof -p, /proc/<pid>/limits, ss -s, systemd limits, and application connection pool metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Question 45Hard

Linux architecture scenario: You need safe connection scaling. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Question 46Medium

Linux security scenario: Raising limits globally hides leaks and increases blast radius. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Question 47Hard

Linux release scenario: You are changing LimitNOFILE and kernel limits. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Question 48Medium

Linux reliability/cost scenario: Connection errors appear during every load test. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Question 49Hard

Linux incident: High connection churn causes port exhaustion. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: ss -s, ephemeral port range, TIME_WAIT counts, conntrack usage, and retry metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Question 50Hard

Linux architecture scenario: You need stable outbound connectivity for high-throughput clients. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Question 51Medium

Linux security scenario: Unsafe sysctl changes can break unrelated workloads. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Question 52Hard

Linux release scenario: You are changing TCP sysctls on production hosts. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Question 53Medium

Linux reliability/cost scenario: NAT and client hosts are saturating ephemeral ports. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Question 54Hard

Linux incident: A deploy fails because the service user cannot read a new config file. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: namei -l, stat, ACL checks, SELinux/AppArmor status, and service user identity. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Question 55Hard

Linux architecture scenario: You need least-privilege filesystem access that survives deploys. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Question 56Medium

Linux security scenario: Using chmod 777 creates a security incident. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Question 57Hard

Linux release scenario: You are standardizing service users and file ownership. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Question 58Medium

Linux reliability/cost scenario: Permission fixes are manual and inconsistent. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Question 59Hard

Linux incident: A process gets permission denied even though Unix permissions look correct. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: ausearch, audit2why, AppArmor logs, context labels, and policy status. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Question 60Hard

Linux architecture scenario: You need mandatory access control without disabling it globally. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Question 61Medium

Linux security scenario: Turning enforcement off hides a real policy violation. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Question 62Hard

Linux release scenario: You are moving from permissive to enforcing mode. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Question 63Medium

Linux reliability/cost scenario: Security requires host hardening before production approval. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Question 64Hard

Linux incident: TLS, logs, and distributed traces disagree because host clocks drift. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: timedatectl, chronyc tracking, NTP source checks, and drift metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Question 65Hard

Linux architecture scenario: You need reliable time across fleets. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Question 66Medium

Linux security scenario: Manual time changes can break databases and distributed locks. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Question 67Hard

Linux release scenario: You are standardizing chrony or systemd-timesyncd. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Question 68Medium

Linux reliability/cost scenario: Debugging incidents is slow because timestamps cannot be trusted. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Question 69Hard

Linux incident: Load average is high but CPU is not fully used. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: uptime, top, vmstat, iostat, blocked process stacks, and run queue analysis. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Question 70Hard

Linux architecture scenario: You need to explain load average correctly in an interview. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Question 71Medium

Linux security scenario: Scaling CPU does not fix IO or uninterruptible sleep. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Question 72Hard

Linux release scenario: You are changing storage or process concurrency. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Question 73Medium

Linux reliability/cost scenario: Executives see high load and ask for bigger instances. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Question 74Hard

Linux incident: Database latency spikes line up with disk await spikes. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: iostat -xz, pidstat -d, filesystem errors, queue depth, and database IO metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Question 75Hard

Linux architecture scenario: You need to separate filesystem, block device, and application causes. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Question 76Medium

Linux security scenario: Filesystem remounts can cause downtime. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Question 77Hard

Linux release scenario: You are changing disk class or mount options. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Question 78Medium

Linux reliability/cost scenario: IOPS costs are high but latency remains unstable. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Question 79Hard

Linux incident: Applications pause for seconds before connecting to dependencies. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: resolvectl, dig +trace, /etc/resolv.conf, packet captures, and resolver logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Question 80Hard

Linux architecture scenario: You need fast and reliable resolver behavior. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Question 81Medium

Linux security scenario: Search domains leak internal names to upstream DNS. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Question 82Hard

Linux release scenario: You are changing resolver config fleet-wide. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Question 83Medium

Linux reliability/cost scenario: Connection latency is dominated by DNS. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Question 84Hard

Linux incident: Zombie or orphaned processes accumulate after deployments. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: ps -eo, parent PID inspection, systemd cgroup view, and signal handling tests. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Question 85Hard

Linux architecture scenario: You need clean process lifecycle management. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Question 86Medium

Linux security scenario: PID exhaustion can take down the host. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Question 87Hard

Linux release scenario: You are adding an init process or fixing signal handling. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Question 88Medium

Linux reliability/cost scenario: Hosts require periodic reboot to stay healthy. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Question 89Hard

Linux incident: A security patch restarts a dependency and causes application errors. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: package changelogs, service restart list, canary host metrics, and maintenance window logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Question 90Hard

Linux architecture scenario: You need safe Linux patching for production fleets. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Question 91Medium

Linux security scenario: Unattended upgrades change runtime behavior without app owners knowing. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Question 92Hard

Linux release scenario: You are introducing rolling OS patch windows. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Question 93Medium

Linux reliability/cost scenario: Vulnerability SLAs require faster patch rollout. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Question 94Hard

Linux incident: A port listens locally but remote clients cannot connect. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: ss -ltnp, iptables/nft rules, cloud firewall rules, route checks, and packet counters. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Question 95Hard

Linux architecture scenario: You need layered host and network firewall troubleshooting. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Question 96Medium

Linux security scenario: Opening broad CIDRs exposes admin services. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Question 97Hard

Linux release scenario: You are migrating iptables rules to nftables. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Question 98Medium

Linux reliability/cost scenario: Network tickets bounce between teams. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Question 99Hard

Linux incident: Backups exist but restore has never been tested. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: backup logs, checksum validation, restore test output, and application consistency checks. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Question 100Hard

Linux architecture scenario: You need provable recovery for Linux-hosted data. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Question 101Medium

Linux security scenario: Backups may contain corrupted or incomplete files. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Question 102Hard

Linux release scenario: You are adding automated restore drills. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Question 103Medium

Linux reliability/cost scenario: Audit requires recovery time and recovery point evidence. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Question 104Hard

Linux incident: An incident has too many logs and no clear timeline. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: journalctl --since, app logs, auth logs, kernel logs, and correlation IDs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Question 105Hard

Linux architecture scenario: You need fast Linux incident reconstruction. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Question 106Medium

Linux security scenario: Logs may include secrets or sensitive request data. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Question 107Hard

Linux release scenario: You are standardizing structured logs and retention. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Question 108Medium

Linux reliability/cost scenario: Mean time to resolution is high. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Question 109Hard

Linux incident: A Linux host fails to boot after a configuration change. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: serial console, rescue mode, journalctl -b -1, fstab validation, and disk checks. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Question 110Hard

Linux architecture scenario: You need a recovery path without data loss. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Question 111Medium

Linux security scenario: Repeated repair attempts can damage disks or overwrite evidence. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Question 112Hard

Linux release scenario: You are changing bootloader, fstab, or kernel settings. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Question 113Medium

Linux reliability/cost scenario: Manual host recovery delays service restoration. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Question 114Hard

Linux incident: A contractor still has SSH access after leaving the project. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and recent change: last, auth logs, authorized keys audit, sudoers review, and identity provider logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Question 115Hard

Linux architecture scenario: You need auditable Linux access management. What design would you choose, and what tradeoffs would you call out in an interview?

#
Show answer

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Question 116Medium

Linux security scenario: Shared users prevent accountability. How do you harden it without breaking production?

#
Show answer

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Question 117Hard

Linux release scenario: You are moving from static SSH keys to centralized access. How do you ship the change safely?

#
Show answer

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Question 118Medium

Linux reliability/cost scenario: Compliance requires proof of least privilege. What signals do you inspect and what changes do you make?

#
Show answer

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Question 119Mediumcgroups

How does a container memory limit actually work under cgroup v2?

#

A container gets OOM-killed even though the app's heap metrics look well under the limit.

Show answer

With cgroup v2, memory.max is the hard limit; when the group cannot reclaim enough below it, the OOM killer kills a process in that group. memory.high is a softer throttle point that forces reclaim and slows the group before the hard limit.

The charged memory includes anonymous memory, page cache, kernel memory such as socket buffers, and tmpfs files. Read memory.current, memory.stat and memory.events (its oom_kill counter) inside the group, and memory.pressure for stalls. A heap that looks small next to a large page cache or tmpfs explains many surprises.

Trade-offs

A tight limit protects neighbours but makes the workload sensitive to cache growth; leave headroom.

Common mistake

Sizing a memory limit from heap metrics alone.

Follow-up

Why might writing large files to an emptyDir with medium Memory trigger an OOM kill?

Question 120HardObservability

How would you find which process makes unexpected outbound connections, with little overhead?

#

The firewall reports connections from a production host to an unknown IP every few minutes.

Show answer

Start with ss -tnp to see current connections and their owning processes. Short-lived connections are often gone before you look, so trace the connect calls instead: eBPF tools such as bcc's tcpconnect or a bpftrace probe report the PID, command and destination for each new TCP connection with low overhead.

For an audit trail, add an auditd rule on the connect syscall. On container hosts, map the PID to its container through its cgroup, and use nsenter to inspect it in its own network namespace. Then confirm what the binary is and how it started.

Trade-offs

eBPF tracing is precise and cheap but needs a reasonably recent kernel and privileges; auditd is older and noisier.

Common mistake

Blocking the IP and moving on without finding the process that made the call.

Follow-up

How would you catch the same behaviour across a whole fleet rather than one host?

Question 121MediumPerformance

What is Pressure Stall Information, and how would you use it?

#

CPU and memory graphs look fine, yet users report slowness during batch windows.

Show answer

PSI reports how much time tasks spend stalled waiting for CPU, memory or I/O, in /proc/pressure/cpu, memory and io, and per cgroup in v2. "some" is the share of time at least one task was stalled; "full" is when all non-idle tasks were stalled at once. Values come as 10-, 60- and 300-second averages plus a running total.

That makes it a better early warning than utilisation: 100% CPU with no pressure is fine, while rising memory pressure means work is waiting on reclaim. Alert on sustained pressure, use per-cgroup values to find the noisy workload, and note that tools like systemd-oomd act on it. Recent Kubernetes versions also expose PSI metrics for nodes and Pods.

Trade-offs

PSI tells you that waiting happens, not why; pair it with per-process tools.

Common mistake

Alerting on high CPU utilisation alone, which pages for healthy busy systems.

Follow-up

What memory pressure level would you treat as an incident for a latency-sensitive service?

Question 122MediumKernel

What is eBPF, and why are networking, security and observability tools moving to it?

#

Your platform team is evaluating Cilium, Falco and an eBPF-based profiler at once.

Show answer

eBPF lets you load small programs into the Linux kernel and attach them to hooks: system calls, kernel and user functions, tracepoints, network packet paths and cgroup events. A verifier checks each program before it runs so it cannot crash the kernel or loop forever, and maps share data with user space.

That lets tools see and act on events where they happen, without kernel modules or sidecars: Cilium does load balancing and network policy, Falco detects suspicious syscalls, profilers sample stacks with low overhead. The costs: you need a reasonably recent kernel with BTF support, privileges to load programs, and care, because a busy probe on a hot path still adds overhead.

Trade-offs

Kernel-level visibility is powerful and efficient, but it is a privileged capability that must be governed.

Common mistake

Granting broad eBPF privileges to any container that asks for them.

Follow-up

How would you check whether your node kernels support the eBPF features a tool needs?

Question 123Hardsystemd

A service has Restart=always, yet after a short database outage it stayed down for hours with "start request repeated too quickly". Why did systemd give up, and how do you design restarts properly?

#

The unit sets only Restart=always. During a 2-minute database outage the service crashed on start every time. When the database came back, the service was still failed.

Show answer

Every start counts toward the unit's start rate limit, including automatic restarts. The defaults are DefaultStartLimitIntervalSec=10s and DefaultStartLimitBurst=5, and RestartSec= defaults to 100ms. A service that crashes immediately is restarted five times in well under a second, hits the limit, and is then no longer restarted. It stays failed until the interval passes and something starts it again (a manual start, a timer or a socket), or until someone runs systemctl reset-failed.

Design the restart behaviour on purpose. Set RestartSec= to a few seconds so restarts do not burn through the burst instantly. Use exponential backoff with RestartSteps= and RestartMaxDelaySec= (for example 10s growing to 160s), so the service keeps retrying during a long outage without hammering anything. Then size StartLimitIntervalSec= and StartLimitBurst= in the [Unit] section to match that schedule, or set the interval to 0 to disable rate limiting for a service that must always come back.

Also fix the root cause: a service that exits when a dependency is down should retry the connection internally, and an OnFailure= unit can alert someone when the unit finally enters the failed state.

Trade-offs

Fast restarts recover quickly from one-off crashes; backoff protects dependencies and avoids hitting the start limit, at the cost of slower recovery.

Common mistake

Putting StartLimitIntervalSec in the [Service] section or assuming Restart=always means systemd will retry forever.

Follow-up

How would you alert on units that have entered the failed state across a fleet?

Backoff restarts that survive a long outage · ini
[Unit]
StartLimitIntervalSec=600
StartLimitBurst=20
OnFailure=notify-oncall@%n.service

[Service]
Restart=on-failure
RestartSec=10s
RestartSteps=4
RestartMaxDelaySec=160s
Question 124Mediumsystemd

Your worker unit has After=api.service, but it still connects before the API is listening, and a typo in ExecStart showed up as a "successful" start. What is going on?

#

Both units use the default Type=simple. The API takes 20 seconds to warm caches before it opens its port.

Show answer

After= orders the start jobs, but it only waits until the earlier unit counts as started, and what started means depends on Type=.

With Type=simple, systemd considers the unit started immediately after it forks the main process, before the binary even runs. So the worker is released straight away, and start commands report success even if the binary cannot be executed, which is why the typo looked fine. Type=exec waits until the binary has actually been executed, so exec failures are reported to systemctl start. That is a cheap improvement for most services.

For real readiness, use Type=notify. The service tells systemd it is ready by sending READY=1 on the notification socket (sd_notify), and units ordered after it wait for that message. Many runtimes have a library for it; for others, a tiny wrapper can send it once the health endpoint answers. Long warm-ups can extend the start timeout with EXTEND_TIMEOUT_USEC=. Avoid Type=forking for new services: the docs discourage it.

Ordering is still not resilience. The worker should retry its connection, because the API can restart at any time after boot.

Trade-offs

Type=notify gives accurate readiness but needs a small code change; Type=exec is free and catches exec failures, but still says nothing about readiness.

Common mistake

Adding ExecStartPost=/bin/sleep 20 to the API instead of signalling readiness.

Follow-up

How does the same readiness problem show up in Kubernetes, and what is the equivalent fix?

api.service: report readiness explicitly · ini
[Service]
Type=notify
ExecStart=/usr/local/bin/api
TimeoutStartSec=120
Question 125Mediumsystemd

You raised nofile to 65536 in /etc/security/limits.conf, rebooted, and the service still fails with "too many open files" at 1,024 connections. Why?

#

ulimit -n in your SSH session shows 65536. cat /proc/<pid>/limits for the service shows a soft limit of 1024.

Show answer

limits.conf is read by the pam_limits module, which applies limits to user login sessions. Your SSH login goes through PAM, so your shell sees 65536. A system service started by systemd is not a login session, so it gets systemd's limits instead. systemd's default DefaultLimitNOFILE= is 1024:524288, a soft limit of 1024 and a hard limit of 524288.

The soft limit stays at 1024 on purpose: select(2) cannot handle file descriptors above 1023, so raising it for every process could break old code. systemd's guidance is that applications that can handle more should raise their own soft limit up to the hard limit. Many modern runtimes do this at startup.

So either fix the app to raise its soft limit, or set it in the unit with a drop-in: LimitNOFILE=65536 under [Service], then systemctl daemon-reload and restart. Verify with /proc/<pid>/limits, never with ulimit in your own shell.

Before raising limits, check for a descriptor leak: if open files grow steadily with constant traffic, a higher limit only delays the outage.

Trade-offs

A per-unit limit is explicit and reviewable; letting the app raise its own soft limit keeps defaults safe for everything else.

Common mistake

Checking ulimit -n in an SSH session and concluding the service has the same limit.

Follow-up

How would you tell a descriptor leak from genuine connection growth?

Drop-in for the unit, then verify on the running process · shell
sudo systemctl edit api.service
# [Service]
# LimitNOFILE=65536
sudo systemctl restart api.service
grep "open files" /proc/$(systemctl show -p MainPID --value api.service)/limits
Question 126Mediumsystemd

You capped a batch job with MemoryMax=4G and it now gets OOM-killed nightly. Someone suggests MemoryHigh= instead. What is the difference, and how would you set both?

#

The job peaks a little above 4 GB for a few minutes while building an index. It shares the host with a latency-sensitive API.

Show answer

MemoryMax= is a hard limit, mapped to cgroup v2 memory.max. If usage cannot be kept under it, the OOM killer is invoked inside the unit, which is what you see every night.

MemoryHigh= is a throttling limit, mapped to memory.high. Above it, the kernel reclaims memory from the unit aggressively and slows it down, but does not kill it. systemd's own recommendation is to use MemoryHigh= as the main control and MemoryMax= as the last line of defence.

For this job: set MemoryHigh=4G so the peak is squeezed rather than fatal, and MemoryMax=5G as a backstop so a real leak still dies before it hurts the API. Protect the API from the other side with MemoryLow= or MemoryMin=; MemoryLow= is weaker protection that may still be reclaimed, MemoryMin= is stronger. If the job writes heavily, IOWeight= (default 100, range 1 to 10000) shares disk bandwidth between units in the same slice.

Watch the job's memory pressure after the change: being throttled for a long time shows up as stall time, and a job that is always throttled needs more memory or a smaller working set.

Trade-offs

Throttling keeps the job alive but makes it slower and less predictable; a hard limit is predictable but turns a small overshoot into a failed run.

Common mistake

Setting MemoryHigh equal to MemoryMax, which gives no room to throttle before the kill.

Follow-up

How would you read the unit's memory pressure from cgroup files to prove the throttling is acceptable?

Throttle first, kill only as a backstop · ini
[Service]
MemoryHigh=4G
MemoryMax=5G
IOWeight=50
Question 127HardMemory

Large builds on a CI host die with SIGKILL, but dmesg shows no kernel OOM killer messages. What else kills processes under memory pressure, and how do you confirm it?

#

Ubuntu build hosts with swap enabled. The whole build container or session disappears at once, not just the biggest process.

Show answer

Look at systemd-oomd. It runs in user space and acts before the kernel's own OOM handling: it watches pressure stall information (PSI) and swap use, and when a configured limit is exceeded it picks a cgroup and sends SIGKILL to all its processes. That matches what you see: no kernel OOM message, and a whole session or service killed at once.

It only acts on units where ManagedOOMSwap= or ManagedOOMMemoryPressure= is set to kill, it needs the unified cgroup v2 hierarchy, memory accounting and a PSI-capable kernel (4.20 or later), and it strongly recommends swap. Check its logs with journalctl -u systemd-oomd, and use oomctl to see which cgroups it monitors and their current pressure.

Then decide the policy, do not just disable it. If builds legitimately use a lot of memory briefly, raise the pressure threshold for the build slice, give builds their own slice with a clear MemoryHigh=, or reduce build parallelism. Test changes with systemd-oomd --dry-run, which logs the kills it would make without making them. Disabling oomd entirely trades occasional killed builds for the risk of the whole host livelocking under memory pressure.

Trade-offs

Early user-space kills keep the host responsive but can kill work that would have finished; relying on the kernel OOM killer kills later and less predictably.

Common mistake

Raising container memory limits because "the build ran out of memory" without seeing that systemd-oomd killed it on pressure.

Follow-up

How would you size swap on build hosts that use systemd-oomd?

Confirm who killed the build · shell
journalctl -u systemd-oomd --since "-2h"
oomctl
dmesg -T | grep -i "out of memory"
Question 128MediumLogging

During an incident, journalctl shows "Suppressed N messages" for your service and the key error lines are missing. What happened, and how do you prevent it?

#

The service logged a burst of retry errors to stdout under systemd. Journald was left on default settings.

Show answer

Journald rate-limits each service. The default is 10,000 messages in 30 seconds (RateLimitBurst= and RateLimitIntervalSec=), scaled up by a multiplier of 1 to 6 based on free disk space. Once a service exceeds it, all further messages in that interval are dropped, and journald writes a notice with the count. The limit is per service, so one noisy service does not eat another's budget.

The right fix is usually the logging, not the limit: a retry loop that logs every attempt should log the first failure, then a summary with counts. If the service legitimately logs at high volume, set LogRateLimitIntervalSec= and LogRateLimitBurst= on that unit rather than raising the global limit.

While you are there, check retention. SystemMaxUse= defaults to 10% of the filesystem, capped at 4G, so a busy host may only keep a few hours. Storage= defaults to persistent in the default namespace, but auto keeps logs only in memory unless /var/log/journal exists, which surprises people on minimal images after a reboot. For anything you need during an incident, ship logs off the host.

Trade-offs

Higher limits keep more evidence but let one noisy service consume disk and journald CPU; lower limits protect the host and lose detail.

Common mistake

Disabling journald rate limiting globally to fix one chatty service.

Follow-up

How would you alert when journald starts suppressing messages for any unit?

Per-unit limit for a known high-volume service · ini
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000
Question 129HardNetworking

Long-lived database connections from private instances fail with "connection reset" after the app is idle for a few minutes, but only for traffic through the NAT gateway. Explain and fix.

#

App servers in a private subnet reach a managed database in another network through an AWS NAT gateway. The connection pool keeps connections open for an hour.

Show answer

Middleboxes keep their own connection state with their own idle timeouts. An AWS NAT gateway times out a connection that is idle for 350 seconds or more, and when either side later tries to use it, the NAT gateway returns an RST (it does not send a FIN). The pool thinks the connection is alive; the first query after a quiet period fails.

Linux TCP keepalive will not save you by default. Keepalive only applies to sockets that enable SO_KEEPALIVE, and the kernel defaults are tcp_keepalive_time of 2 hours, tcp_keepalive_intvl of 75 seconds and 9 probes, so the first probe goes out long after the NAT has forgotten the flow.

Fix it at the right layer. Enable keepalive in the client or driver with an idle time below the shortest middlebox timeout, for example 60 to 120 seconds; most database drivers and pools have a setting. Also set the pool's maximum idle time or connection lifetime below 350 seconds so idle connections are closed cleanly, and validate connections on borrow. Changing the host-wide net.ipv4.tcp_keepalive_time helps only sockets that already enable keepalive, and it affects every application on the host.

Trade-offs

Short keepalives keep flows alive through middleboxes at the cost of a little background traffic; short pool lifetimes avoid stale connections but reconnect more often.

Common mistake

Tuning tcp_keepalive_time on the host and assuming every connection now sends keepalives.

Follow-up

Which other middleboxes in a typical cloud path have idle timeouts you should know?

Check the kernel defaults and which sockets use keepalive · shell
sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
ss -tno state established | grep keepalive
Question 130HardNetworking

A service that calls one upstream at high rate fails with "cannot assign requested address". A colleague sets tcp_tw_reuse=1 and someone else suggests tcp_tw_recycle. What do you actually do?

#

The client opens a new TCP connection per request to a single upstream IP and port. ss -s shows tens of thousands of sockets in TIME-WAIT.

Show answer

Each outbound connection to the same destination IP and port needs a unique local port. The default ip_local_port_range is 32768 to 60999, roughly 28,000 ports. The side that closes a connection keeps it in TIME-WAIT for a while, so at a few hundred new connections per second to one upstream you run out of local ports, and connect() fails.

The real fix is to stop opening a connection per request: use HTTP keep-alive or a connection pool, and the problem disappears along with the handshake latency. If you still need more headroom, widen the local port range, or spread traffic over more destination IPs or source IPs.

On the sysctl: tcp_tw_reuse allows reuse of TIME-WAIT sockets for new connections when it is safe from the protocol's point of view. The default is 2, which enables it for loopback traffic only; 1 enables it globally. Setting 1 is a reasonable, low-risk addition for a client with this pattern, but it is a mitigation, not the fix. Do not use tcp_tw_recycle: AWS documents it on remote servers as a known cause of failed connections from clients behind a NAT, and recommends tcp_tw_reuse as the safer option.

Trade-offs

Connection reuse fixes the cause and lowers latency, but needs client changes and care with idle timeouts; sysctl changes are quick but host-wide and only buy headroom.

Common mistake

Raising sysctls on every host instead of fixing the one client that opens a connection per request.

Follow-up

Why does TIME-WAIT exist at all, and what could go wrong without it?

Confirm the pattern before tuning · shell
ss -s
ss -tan state time-wait dst 10.0.4.20 | wc -l
sysctl net.ipv4.ip_local_port_range net.ipv4.tcp_tw_reuse
Question 131MediumNetworking

Under traffic bursts, clients see connection timeouts to a service, but its CPU is low and its own logs show nothing. How do you check whether the accept queue is overflowing?

#

The service calls listen(fd, 65535) and runs a single accept loop. It runs on a mix of old and new kernels.

Show answer

For TCP, the listen() backlog is the queue of fully established connections waiting for the application to call accept(). If the app accepts more slowly than connections arrive, the queue fills, and new connections are dropped or time out before the app ever sees them, so the app logs nothing.

The backlog you pass is silently capped to net.core.somaxconn. Since Linux 5.4 the default is 4096; on earlier kernels it is 128. So listen(fd, 65535) gives 128 on an old kernel, which explains why only some hosts are affected. The incomplete (SYN) queue is separate, sized by tcp_max_syn_backlog, and ignored when SYN cookies are enabled.

Check it directly: ss -lnt shows, for listening sockets, the current accept queue in Recv-Q and the configured backlog in Send-Q. A Recv-Q at or near Send-Q during bursts is your answer. The kernel's TCP counters (nstat -az | grep -i listen) show overflows and drops since boot.

Fix the slow accept loop first: more acceptor threads, SO_REUSEPORT with several listeners, or moving work off the accept path. Then raise somaxconn on old kernels so the backlog the app asks for actually applies.

Trade-offs

A bigger queue absorbs bursts but adds queueing latency and hides an accept loop that cannot keep up.

Common mistake

Raising the backlog in the app config without checking that somaxconn silently capped it.

Follow-up

How does a load balancer's health check behave when the accept queue is full?

Is the accept queue full? · shell
ss -lnt "sport = :8080"
sysctl net.core.somaxconn
nstat -az | grep -i listen
Question 132HardNetworking

A busy NAT or Kubernetes node starts dropping packets at random, and the kernel log says "nf_conntrack: table full, dropping packet". What is happening and what do you change?

#

The node handles many short-lived connections plus some long-lived idle ones. Nobody has tuned netfilter settings.

Show answer

Netfilter connection tracking keeps an entry for every flow it tracks, which NAT, stateful firewall rules and kube-proxy all rely on. When nf_conntrack_count reaches nf_conntrack_max, new flows cannot be tracked and their packets are dropped, which looks like random connection failures.

The defaults are sized from memory: the hash table size (nf_conntrack_buckets) is total memory divided by 16384, and nf_conntrack_max is set to the bucket count by default. So a small node gets a small table. And nf_conntrack_tcp_timeout_established defaults to 432000 seconds, five days, so established connections that were never closed cleanly can hold entries for a very long time.

Recover by raising nf_conntrack_max (and the hash size to match, so lookups stay fast), checking memory headroom first. Then look at what fills the table with conntrack -S and conntrack -L counts by state. Lower the established timeout to something closer to your real idle timeouts, and reduce churn by reusing connections. Alert on nf_conntrack_count as a percentage of nf_conntrack_max, so you see the next one coming.

Trade-offs

A larger table costs kernel memory; shorter timeouts free entries sooner but can break very long idle connections that still matter.

Common mistake

Raising nf_conntrack_max without the hash size, which makes every lookup slower.

Follow-up

Which traffic could you exclude from connection tracking entirely, and what would you lose?

How full is the table and what is in it? · shell
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_buckets
conntrack -S
conntrack -L -p tcp 2>/dev/null | awk "{print \$4}" | sort | uniq -c | sort -rn
Question 133HardMemory

A latency-sensitive in-memory store shows occasional multi-millisecond stalls that line up with high system CPU, not with GC or I/O. How would you test whether Transparent Huge Pages are involved?

#

The store allocates large heaps. Hosts run a stock distribution kernel with default THP settings.

Show answer

Transparent Huge Pages back anonymous memory with huge pages where possible. The cost appears when the kernel has to create one at fault time. With the defrag setting at always, an allocation that cannot find a huge page stalls while the kernel directly reclaims pages and compacts memory. With madvise (the kernel's documented default for defrag), only regions that asked for huge pages with madvise(MADV_HUGEPAGE) stall that way. defer never stalls the faulting process: it wakes background reclaim and compaction, and khugepaged installs huge pages later. The kernel docs also warn that with THP enabled system-wide, applications can end up allocating more memory.

Test instead of guessing. Check /sys/kernel/mm/transparent_hugepage/enabled and defrag, watch thp_fault_alloc and thp_fault_fallback in /proc/vmstat, and compare AnonHugePages in /proc/meminfo. Correlate compaction counters and stalls with your latency spikes, and profile system CPU during a stall.

If THP is the cause, prefer enabled=madvise with defrag=defer or madvise, so only applications that ask for huge pages get them and nobody stalls on compaction by default. Follow the store's own documented recommendation too: Redis's admin guide, for example, says to set THP enabled to never so it does not hurt Redis memory usage and latency.

Trade-offs

Huge pages cut TLB misses and help many large-memory workloads; synchronous defrag and memory bloat can hurt latency-sensitive ones.

Common mistake

Disabling THP fleet-wide from a blog post without measuring, and slowing down workloads that benefited from it.

Follow-up

How would you make the THP setting persistent and consistent across a fleet?

Current THP mode and fault behaviour · shell
cat /sys/kernel/mm/transparent_hugepage/enabled /sys/kernel/mm/transparent_hugepage/defrag
grep -E "thp_fault_alloc|thp_fault_fallback|compact_stall" /proc/vmstat
grep AnonHugePages /proc/meminfo
Question 134HardStorage

Every night a large backup copy on a database host makes the database's own small writes stall for seconds. Disk throughput is not maxed. What is happening, and how do you contain it?

#

The host has 256 GB of RAM. The backup streams hundreds of GB to a local disk with plain cp. Dirty page settings are at distribution defaults.

Show answer

Buffered writes land in the page cache as dirty pages. Background flusher threads start writing them out once dirty data passes dirty_background_ratio; when it passes dirty_ratio, a process that is generating writes is made to start writing out dirty data itself. Both are percentages of available memory, so on a 256 GB host the thresholds are huge: the backup builds up a vast amount of dirty data, then the flush storm and throttling hit everyone writing to that device, including the database's latency-sensitive writes and fsyncs.

Contain it in layers. Use dirty_background_bytes and dirty_bytes instead of ratios on large-memory hosts, so the thresholds are absolute and much smaller; each bytes setting replaces its ratio counterpart. Make the backup a good neighbour: run it in its own systemd unit or cgroup with a low IOWeight= (or an I/O bandwidth limit), and consider direct I/O or a tool that rate-limits. Best of all, move backups off the database's data disk or use the database's own backup tooling with throttling.

Measure with grep -E 'Dirty|Writeback' /proc/meminfo during the backup alongside database write latency, so you can show the cause and the effect.

Trade-offs

Small dirty thresholds smooth latency but reduce write batching and throughput for bulk jobs; cgroup I/O weights only work when the device and scheduler support them.

Common mistake

Lowering dirty_ratio host-wide without measuring, and slowing every bulk write on the machine.

Follow-up

Why does an fsync from the database get slower when another process has lots of dirty pages on the same filesystem?

Run the backup as a low-priority I/O unit · ini
[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
IOWeight=10
Nice=10
Question 135MediumMemory

After a painful OOM incident, someone proposes setting vm.overcommit_memory=2 on all hosts "so the OOM killer never runs again". What do you tell them?

#

Hosts run JVM services, a Redis instance and several Python workers. The OOM incident was caused by one leaking worker.

Show answer

The default, overcommit_memory=0, uses a heuristic: it compares a request against total memory plus swap and refuses only obvious overcommits. Mode 1 pretends there is always enough memory until it actually runs out. Mode 2 is the never-overcommit policy: it tries to prevent any overcommit at all.

Mode 2 does not make memory problems go away; it moves them. Allocations start failing early, inside applications, whenever committed memory hits the limit, even if most of that memory is reserved but never touched. Runtimes that reserve large virtual ranges up front, and processes that fork with a large address space, can fail with allocation errors on a host that has plenty of free RAM. Some software documents the opposite recommendation: Redis's admin guide tells you to set vm.overcommit_memory = 1, pointing to background saves that fail with a fork() error. Few applications handle a failed allocation gracefully, so you trade a clear OOM kill for random crashes in random processes.

Fix the actual incident instead: a memory limit on the leaking worker's unit or container (MemoryHigh= plus MemoryMax=), oom_score_adj or protection settings so critical services are chosen last, and alerting on memory pressure before it becomes an OOM.

Trade-offs

Strict accounting gives predictable commit limits, useful on a few dedicated systems; it costs much lower usable memory and brittle failures for general workloads.

Common mistake

Changing a host-wide kernel policy in response to a single leaking process.

Follow-up

How do cgroup memory limits and the global overcommit policy interact on a container host?

Question 136MediumBoot

A cloud VM dropped into emergency mode after reboot because a data volume was detached. How do you write fstab entries so a missing data disk cannot stop the boot?

#

fstab has /dev/nvme1n1 /data xfs defaults 0 2. The volume was detached for a snapshot restore and the instance was rebooted.

Show answer

systemd turns each fstab line into a mount unit. Without options, a local mount is required by local-fs.target, so if its device never shows up, the target fails and the boot drops into emergency mode, with no SSH on a cloud VM.

For anything that is not needed to boot the OS, add nofail: the mount is then only wanted, not required, by local-fs.target, and the boot does not wait for it. Add x-systemd.device-timeout= to shorten how long systemd waits for the device to appear. Mount by UUID= or a stable label, not /dev/nvme1n1; NVMe device names can change order between boots and instance types. For network file systems, _netdev marks the mount as needing the network so it is ordered after the network is online. And x-systemd.automount creates an automount unit, so the file system is mounted on first access, which keeps slow or optional storage off the boot path entirely.

Then make the dependency explicit: services that need /data should declare RequiresMountsFor=/data, so they fail clearly instead of writing into an empty mount point on the root disk.

Trade-offs

nofail keeps the OS reachable, but services can start without their data unless they declare the mount dependency.

Common mistake

Adding nofail and forgetting RequiresMountsFor, so the app fills the root disk writing to the empty /data directory.

Follow-up

How would you recover an instance that is already stuck in emergency mode without console access?

/etc/fstab entry for an optional data disk · text
UUID=3f1c0b6e-8d7a-4c2e-9a51-2b7e4f0d9c11  /data  xfs  defaults,nofail,x-systemd.device-timeout=10s  0  2
Question 137HardSecurity

After moving CI runners to Ubuntu 24.04, rootless container builds and headless Chrome tests fail with permission errors creating namespaces. Why, and what is the right fix?

#

The same jobs worked on Ubuntu 22.04. Someone has already proposed setting kernel.apparmor_restrict_unprivileged_userns=0 in the runner image.

Show answer

Unprivileged user namespaces are a large kernel attack surface, so Ubuntu restricts them through AppArmor. The feature arrived in Ubuntu 23.10 as opt-in; in 24.04 the release notes state that the Ubuntu kernel now restricts unprivileged user namespaces in combination with the apparmor package. Under the restriction, an unprivileged process may create a user namespace only if it is confined by an AppArmor profile that has a userns rule (or holds CAP_SYS_ADMIN). The default profile lets unconfined applications create the namespace but denies them any capabilities inside it, which is exactly what rootless builders and Chrome's sandbox need, so they fail.

The right fix is a profile, not turning the protection off. Ubuntu ships profiles for common applications, and for software that is hard to confine there is a default_allow profile mode: the program stays unconfined but gets the userns rule. Software installed at an unexpected path, or with no profile, is still blocked, which is why a browser in a custom CI image often breaks while the packaged one works. Write a small profile for the exact binary path your jobs run, and ship it with the runner image.

Setting the sysctls to 0 re-enables unprivileged namespaces for every process on the runner. On CI hosts that run untrusted pull-request code, that is the wrong trade.

Trade-offs

Per-binary profiles keep the protection for everything else but must track install paths; disabling the restriction is easy and widens the attack surface for all code on the host.

Common mistake

Baking kernel.apparmor_restrict_unprivileged_userns=0 into every runner image.

Follow-up

Why are unprivileged user namespaces such a common ingredient in Linux privilege escalation exploits?

Example profile from Ubuntu's announcement: userns for one binary path · text
abi <abi/4.0>,

include <tunables/global>

/opt/google/chrome/chrome flags=(default_allow) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/opt.google.chrome.chrome>
}
Question 138MediumSecurity

After upgrading your bastion hosts to OpenSSH 10, some old automation and network devices can no longer connect, and one inventory script reports the wrong version. What changed?

#

A mixed estate: modern laptops, a few appliances with old SSH stacks, and scripts that parse SSH version banners.

Show answer

OpenSSH 10.0 (released 2025-04-09) has several changes that break old peers. It removes the DSA signature algorithm entirely, so DSA host or user keys stop working. sshd disables finite-field (modp) Diffie-Hellman key exchange by default, which some old clients and appliances rely on. The default key agreement is now the hybrid post-quantum mlkem768x25519-sha256, which is fine for modern peers. And the version string is now SSH-2.0-OpenSSH_10.0, which confuses software that matches on patterns like OpenSSH_1* and thinks it is talking to version 1.x.

Later releases add more: OpenSSH 10.1 warns when a connection negotiates a key agreement that is not post-quantum (the WarnWeakCrypto option, on by default), and moves agent sockets from /tmp to under ~/.ssh/agent, which can break scripts that look for the old path.

Handle it as a planned migration. Inventory keys and peers first (any DSA keys must be replaced, ideally with Ed25519). For appliances that cannot be upgraded, keep a separate, tightly firewalled legacy jump path for just those devices, with an owner and an end date, rather than re-enabling weak algorithms on the main bastion. Fix version parsing to compare numbers, not string prefixes.

Trade-offs

A separate legacy path keeps old equipment working without weakening every connection, but each one is debt that needs an owner and a removal date.

Common mistake

Rolling back the bastion upgrade, which also removes the security fixes that came with it.

Follow-up

How would you detect, from server logs, which clients still negotiate non-post-quantum key exchange?

Question 139MediumPerformance

Your CPU flame graphs from perf show shallow, broken stacks full of unknown frames, so nobody trusts them. Why, and what changed on newer Ubuntu releases?

#

You profile C and Go services on a mix of Ubuntu 22.04 and 24.04 hosts with perf record -g.

Show answer

To build a call stack, the profiler has to walk from each frame to its caller. The cheap way is the frame pointer register. Code compiled without frame pointers forces tools to use DWARF unwind information instead, which is significantly more expensive and often impractical in production, so you get truncated stacks or unknown frames.

Starting with Ubuntu 24.04 LTS, Canonical's GCC enables frame pointers by default on 64-bit platforms, so distribution packages are built with them and tools like perf, bpftrace and bcc-tools work out of the box. Canonical measured the cost at between 1% and 2% in most cases. It is not universal: 32-bit architectures were excluded, and some packages, such as the Python interpreter, were excluded because the impact was high. Your own binaries still depend on your build flags.

So on 22.04 hosts, system libraries break your stacks; on 24.04 they mostly do not. For your own C and C++ code, build with -fno-omit-frame-pointer. For interpreted runtimes, use the runtime's own profiling support. Then the profile becomes evidence you can act on, including off-CPU profiling of where threads wait.

Trade-offs

Frame pointers cost a percent or two of CPU and give cheap, always-on profiling; omitting them saves that but makes production profiling unreliable.

Common mistake

Switching every profile to DWARF unwinding in production and paying a large overhead to work around missing frame pointers.

Follow-up

What is off-CPU profiling, and which kind of latency problem does it find that CPU flame graphs miss?

Frame-pointer based call graph, then fall back to DWARF for one process · shell
perf record -F 99 -g -p $PID -- sleep 30
perf record -F 99 --call-graph dwarf -p $PID -- sleep 10
Question 140EasyFilesystems

A file watcher fails with "No space left on device", but df shows plenty of free space and inodes. What is the real limit?

#

A developer VM runs several IDEs, a log shipper and a hot-reload dev server, all watching large source trees.

Show answer

inotify_add_watch() returns ENOSPC when the user limit on the total number of inotify watches is reached (or the kernel cannot allocate a needed resource). Same error code as a full disk, completely different cause.

The limit is /proc/sys/fs/inotify/max_user_watches, and it is per real user ID: every watcher running as the same user shares it, so one IDE indexing node_modules can break the log shipper. There is a separate max_user_instances limit on inotify instances per user, which gives a different set of errors when exceeded. If a watcher falls behind, events beyond max_queued_events are dropped and it receives an IN_Q_OVERFLOW event, after which it must rescan.

Fix: exclude generated directories from watchers (node_modules, build output, .git), find the heavy watcher by counting inotify file descriptors per process under /proc/*/fdinfo, and raise fs.inotify.max_user_watches with a sysctl.d file if the workload genuinely needs it. Each watch uses kernel memory, so do not set it to an arbitrary huge number on small machines.

Trade-offs

Higher limits remove the error but cost kernel memory per watch; excluding paths fixes the cause and makes watchers faster.

Common mistake

Deleting files to free disk space in response to an inotify ENOSPC.

Follow-up

Why can a watcher that hits IN_Q_OVERFLOW miss file changes, and how should it recover?

Check limits and persist a higher watch limit · shell
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
echo "fs.inotify.max_user_watches=524288" | sudo tee /etc/sysctl.d/60-inotify.conf
sudo sysctl --system
Question 141Mediumsystemd

You hardened a unit with ProtectSystem=strict, ProtectHome=true and NoNewPrivileges=true, and now the app cannot write its data, and a helper that uses sudo stops working. How do you keep the hardening and fix both?

#

The app writes to /var/lib/myapp and /var/log/myapp. A cleanup step in ExecStartPost calls sudo to rotate a file.

Show answer

ProtectSystem=strict mounts the entire file system hierarchy read-only for the unit, except the API file systems such as /proc and /sys. ProtectHome=true makes /home, /root and /run/user inaccessible and empty. So the app's writes fail by design.

Do not punch generic holes with ReadWritePaths= first. Declare what the service owns: StateDirectory=myapp gives it /var/lib/myapp, LogsDirectory=myapp gives /var/log/myapp, and RuntimeDirectory= and CacheDirectory= work the same way. systemd creates them, makes them writable for the unit even under strict protection, and makes the innermost directory owned by the unit's User= and Group=. State, cache, logs and configuration directories survive a stop; runtime directories are removed when the unit stops unless you preserve them. These also work with DynamicUser=, where systemd allocates a user and group when the unit starts and releases them when it stops.

NoNewPrivileges=true ensures the process and its children can never gain privileges through execve, so sudo and setuid helpers cannot elevate. That is the point. Move the privileged step out: give the unit the directory it needs, or run the rotation as its own small unit with exactly the rights it needs.

Trade-offs

Declared directories keep strict protection and document what the service writes; ReadWritePaths is quicker but grows into an unreviewed allowlist.

Common mistake

Removing ProtectSystem because one path was not writable.

Follow-up

How would you score and compare the exposure of all services on a host after hardening?

Strict protection with declared writable directories · ini
[Service]
User=myapp
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true
PrivateTmp=true
StateDirectory=myapp
LogsDirectory=myapp
Question 142Easysystemd

An app writes an export to /tmp, but the operator cannot find the file in /tmp on the host, and it disappears after a restart. Where did it go?

#

The app runs as a systemd service with PrivateTmp=true, added by a security baseline.

Show answer

With PrivateTmp= enabled, systemd sets up a new file system namespace for the service with its own private /tmp and /var/tmp. Your shell sees the host's /tmp; the service sees its own. The private directories belong to the unit's lifetime, so they are cleaned up when the service stops, which is why the file vanished on restart. With PrivateTmp=disconnected the directories are backed by a new tmpfs that is disconnected from the host's storage.

That isolation is a good default: other users and services cannot read or tamper with the app's temporary files, a classic source of race and symlink attacks.

The fix is not to turn it off. Output that someone needs after the process exits is not temporary; write it to a declared directory such as StateDirectory= or CacheDirectory=, or straight to object storage. If you need to inspect the private /tmp of a running service, enter its mount namespace with nsenter using the service's main PID.

Trade-offs

Private /tmp blocks cross-service tampering and leaks, but surprises operators who expect a shared /tmp.

Common mistake

Disabling PrivateTmp across the baseline to make one export visible.

Follow-up

Which other systemd sandboxing options change what paths a service can see?

Question 143HardNetworking

You added a second network interface to a host. Traffic arriving on it is silently dropped, even though tcpdump shows the packets arriving. What do you check?

#

eth0 has the default route. eth1 is on a storage network. Clients on a third network reach the host via eth1, but replies would leave via eth0.

Show answer

This is reverse path filtering. With rp_filter in strict mode (1, RFC 3704 strict reverse path), the kernel drops a packet if the best route back to its source does not go out of the interface it arrived on. Here, the route back to the clients is via eth0's default route, so packets arriving on eth1 fail the check and are dropped after tcpdump has already captured them, which makes it confusing. Mode 2 is loose: the source only has to be reachable through some interface. Mode 0 disables the check. The kernel's own default is 0, but distributions and images often set it, so check the actual values (the effective setting is the maximum of the all and per-interface values).

The right fix is usually routing, not disabling the filter. Use policy routing: a separate routing table for eth1 with its own default gateway, and an ip rule that uses that table for traffic sourced from eth1's address. Then replies leave the way requests came in, the strict check passes, and you avoid asymmetric paths that stateful firewalls also drop. If you cannot fix routing, set loose mode on that interface rather than turning the check off.

Confirm with ip route get <client-ip> from <eth1-ip> iif eth1 and by watching drop counters while testing.

Trade-offs

Policy routing is correct and keeps spoofing protection, but adds configuration that must be persisted; loose mode is quick but weaker against spoofed sources.

Common mistake

Setting rp_filter=0 on all interfaces and leaving asymmetric routing in place.

Follow-up

How does reverse path filtering interact with Kubernetes CNIs that use multiple interfaces?

Policy routing so replies leave via eth1 · shell
sysctl net.ipv4.conf.all.rp_filter net.ipv4.conf.eth1.rp_filter
ip route add default via 10.50.0.1 dev eth1 table 100
ip rule add from 10.50.0.10/32 table 100
ip route get 192.0.2.25 from 10.50.0.10 iif eth1
Question 144HardNetworking

Over a new VPN link, SSH and small API calls work, but large HTTPS downloads and git clone hang forever. How do you prove it is an MTU black hole and fix it?

#

The tunnel adds encapsulation overhead. A firewall in the path drops all ICMP. Hosts use a 1500-byte MTU.

Show answer

Path MTU discovery depends on routers sending ICMP "fragmentation needed" messages back when a packet is too big for the next hop. The tunnel's overhead lowers the path MTU below 1500, and the firewall drops the ICMP, so the sender never learns to send smaller segments. The handshake and small requests fit; the first full-size data segment is dropped silently, and the transfer hangs. That pattern, small works and large hangs, is the signature.

Prove it with a ping that forbids fragmentation and increases the size until it fails: ping -M do -s 1372 host succeeds while -s 1472 fails, for example. tracepath also reports the discovered path MTU, and a capture shows retransmissions of full-size segments with no ICMP coming back.

Fix it in order of preference: allow the ICMP messages PMTUD needs through the firewall; clamp the TCP MSS on the tunnel gateway to fit the tunnel MTU, which fixes TCP for every host behind it; or set the correct MTU on the tunnel interface. On hosts, net.ipv4.tcp_mtu_probing=1 enables packetization-layer probing only when an ICMP black hole is detected, a useful safety net. Remember UDP-based protocols do not benefit from MSS clamping.

Trade-offs

MSS clamping fixes TCP centrally and quickly but hides the broken ICMP policy; allowing the right ICMP types fixes the cause for all protocols.

Common mistake

Blocking all ICMP for "security", which breaks path MTU discovery.

Follow-up

Why do container overlay networks often need a smaller MTU than the host interface?

Find the largest packet that passes unfragmented · shell
ping -c 3 -M do -s 1472 10.60.0.5   # 1500-byte packet
ping -c 3 -M do -s 1372 10.60.0.5   # 1400-byte packet
tracepath -n 10.60.0.5
Question 145MediumPatching

After moving to Ubuntu 24.04, an unattended security update restarted a stateful service mid-day without anyone deploying anything. Why, and how do you control it?

#

Unattended upgrades install security updates daily. A library update was applied at 13:00 and the service restarted seconds later, dropping connections.

Show answer

Ubuntu 24.04 changed needrestart to systematically restart services affected by a library upgrade, including during unattended upgrades. That is good security behaviour: a patched library on disk does nothing for a process that still has the old version mapped in memory. But it means patching now restarts services, not just files on disk.

Control it deliberately rather than turning it off. Schedule unattended upgrades inside a maintenance window for stateful hosts, so the restart happens when you expect it. Configure needrestart to list rather than restart for the few services that need a controlled, drained restart, and run those restarts from your own automation, for example one node at a time with traffic drained. Make the services themselves restart-safe with graceful shutdown and fast startup, since you will restart them for patches anyway.

The risky outcome is the opposite one: disabling restarts entirely leaves vulnerable library code running for weeks while your scanner reports the host as patched.

Trade-offs

Automatic restarts close the patch gap quickly at the cost of surprise restarts; controlled restarts are safer for stateful systems but need tooling and discipline to happen at all.

Common mistake

Disabling needrestart globally and assuming installed packages mean the running processes are patched.

Follow-up

How would you find processes still running with deleted or replaced library files after an update?

Question 146MediumMemory

What is the OOM killer and how does it choose a victim?

#

A host kills a long-running batch job under memory pressure while a smaller but important service survives.

Show answer

When the kernel cannot reclaim enough memory to satisfy an allocation, it invokes the OOM killer. Each candidate gets a badness score based largely on how much of its allowed memory it is using; root processes get a small allowance bonus. The process with the highest effective score is killed, so a big memory hog is usually the target.

oom_score_adj shifts that score: it accepts -1000 (badness zero, effectively never chosen) through +1000 (always chosen). You can read /proc/<pid>/oom_score and adjust oom_score_adj with choom, which is the practical way to protect or sacrifice a specific process.

The scope depends on the limit that was hit. With cgroup v2 and memory.max, the killer acts inside that cgroup, and memory.oom.group=1 kills the whole workload as a unit. systemd-oomd is separate: it is userspace, uses PSI pressure information, and kills processes in managed cgroups before the kernel runs out of memory.

Trade-offs

Protecting a process with -1000 does not protect the rest of its cgroup or the host from memory exhaustion.

Common mistake

Confusing systemd-oomd with the kernel OOM killer, or assuming the oldest process is always killed.

Follow-up

How would you find out after the fact which cgroup triggered the kill?

Question 147EasyPerformance

What is CPU steal time and iowait, and what do they indicate?

#

A VM is slow, but its application CPU usage looks low. mpstat shows a high %steal and a rising %iowait.

Show answer

%iowait is the percentage of time a CPU was idle while the system still had an outstanding disk I/O request. It points at storage latency or blocked I/O, not at CPU saturation; a high value alone does not prove the disk is the only problem.

%steal is the time a virtual CPU spent involuntarily waiting while the hypervisor serviced another virtual processor. It exists only in virtualised environments and usually means the host is oversubscribed: your VM is ready to run but cannot get a physical CPU. The remedy sits with the host or instance sizing, not with optimising application code.

Both appear as separate columns in mpstat (%iowait, %steal) and as wa and st in vmstat. Read them alongside run queue length, disk latency and hypervisor metrics instead of treating either as a standalone bottleneck.

Trade-offs

Adding vCPUs can lower steal on a contended host, but it may increase cost without helping a single-threaded workload.

Common mistake

Calling iowait "busy CPU time" or looking for steal time on bare metal.

Follow-up

What would you check next if %iowait is high but device latency looks normal?

Question 148MediumNetworking

What is the order of name resolution on Linux (nsswitch.conf)?

#

A hostname in /etc/hosts is being ignored on one host, and single-label names resolve differently between two servers.

Show answer

/etc/nsswitch.conf defines a source list per database. A common line is hosts: files mdns4_minimal [NOTFOUND=return] dns, meaning: try /etc/hosts, then mDNS, stop if that source says the name does not exist, otherwise query DNS. Status actions such as [NOTFOUND=return] and [!UNAVAIL=return] let you control when lookup stops or continues. If files comes after dns, an /etc/hosts entry can be shadowed.

DNS behaviour itself comes from /etc/resolv.conf: nameserver, search domains and options ndots. With ndots:5, a name containing fewer than five dots is tried with search suffixes before the absolute name.

On systemd-resolved hosts, /etc/resolv.conf commonly points at the local stub 127.0.0.53, and resolved handles the upstream servers; 127.0.0.54 is the proxy-only stub. When nss-resolve is enabled it replaces both files and dns for hosts, and its compatibility notes say names containing a dot are treated as FQDNs and ndots is not implemented.

Trade-offs

Long search lists make short names convenient but can turn a typo into many failing DNS queries.

Common mistake

Editing /etc/hosts and assuming it wins regardless of the nsswitch order, or quoting a 127.0.0.55 stub.

Follow-up

How would you trace exactly which source answered a lookup?