Linux
Think it through.

A hard link is another directory entry pointing to the same inode — the file exists as long as any hard link remains, and all links are equal. A symlink is a small file that stores a path to another file; it can cross filesystems and link directories, but breaks if the target is moved or deleted. Hard links can't cross filesystems or point to directories.

Use ss -ltnp 'sport = :8080' (or lsof -i :8080, or older netstat -ltnp) to list the listening socket and its PID/program. Then inspect or kill it with ps -p <pid> / kill. ss is the modern, faster replacement for netstat.

Start broad with top/htop and uptime (load average vs core count). Then split by resource: CPU (top, pidstat), memory (free -h, check swap and OOM in dmesg), disk I/O (iostat -x, iotop), and network (ss, iftop). vmstat 1 gives a quick overall view. The goal is to localize the bottleneck to CPU, memory, I/O, or network before drilling into the offending process or query.

755 = owner rwx (7), group r-x (5), others r-x (5) — common for executables and directories. SUID (e.g. 4755, the leading 4) makes an executable run with the file owner's privileges rather than the caller's — that's how passwd edits /etc/shadow as root. SUID on binaries is a security-sensitive setting and a common privilege-escalation vector.

SIGTERM (15) politely asks a process to terminate — it can catch it, clean up, and exit; this is what you should use first and what kill sends by default. SIGKILL (9) cannot be caught or ignored; the kernel kills the process immediately with no cleanup — last resort. SIGHUP originally meant terminal hangup but by convention tells daemons to reload their config without restarting. Graceful shutdown handlers in apps/containers hook SIGTERM.

free -h reads from /proc/meminfo, which shows the host's global memory statistics, not the container's cgroup limits. In cgroup v2, the container's memory limit is enforced at the cgroup level, not via proc. The correct way to see container memory usage is cat /sys/fs/cgroup/memory.current and cat /sys/fs/cgroup/memory.max. The 4GB shown by free is the host's total memory. cgroup v2 memory accounting includes: memory.current (total page cache + RSS + swap), memory.swap.current (swap usage), and memory.stat for detailed breakdown (anon, file, kernel, slab). Unlike cgroup v1, v2 unifies memory and memsw accounting — the memory.max limit applies to memory+swap combined. The kernel enforces OOM via memory.events — when max is exceeded, oom_kill increments. If free shows 4GB, the process is seeing host memory. For accurate container-level memory, always read the cgroup files. Docker's docker stats reads these cgroup values correctly.

The default TimeoutStartSec is 90 seconds (systemd v240+). When the script exceeds this, systemd sends SIGTERM (configurable via KillSignal), then SIGKILL after TimeoutStopSec. To fix: override the timeout in a drop-in via systemctl edit myservice.service and add [Service] TimeoutStartSec=300 (or infinity). However, if the script hangs indefinitely, this masks the problem. Better: use Type=notify with sd_notify("READY=1") in the script, so systemd knows the service is alive without waiting for the full timeout. If the script is not modifiable, use Type=simple with ExecStartPost that polls for readiness — or use Restart=on-failure with RestartSec=30 to handle transient delays. Also consider TimeoutStartSec behavior differs between Type=simple (systemd waits for the execve only) and Type=oneshot or Type=notify (waits for process exit or notification). For long initialization, Type=idle delays start until boot completes.

futex(FUTEX_WAIT_PRIVATE) means the thread is blocked waiting on a userspace mutex. The futex syscall is the kernel-level wait operation — when FUTEX_WAIT_PRIVATE returns EWOULDBLOCK, the thread re-checks the userspace lock word; if still contended, it sleeps. The address 0x7f... is the userspace address of the futex word (part of the pthread_mutex_t or pthread_cond_t). To identify the contended mutex: (1) Check /proc/<PID>/maps to find which shared library or heap region maps that address. (2) Run gdb -p <PID>, then info threads, thread apply all bt to see which thread holds the lock. (3) Use futex traces from all threads — the lock owner is not in FUTEX_WAIT but in FUTEX_WAKE or running code. (4) Enable lockdep with echo 1 > /sys/kernel/debug/tracing/events/lock/lock_acquire/enable. (5) Use perf lock record -p <PID> then perf lock report to show contended locks by symbol. For Go programs, use GOTRACEBACK=all or the runtime/trace output instead.

User namespaces map a range of UIDs inside the namespace to a different range on the host via /proc/<PID>/uid_map. For example, 0 100000 65536 maps container UID 0 to host UID 100000. This means the container's root has no special privileges on the host — kernel capabilities are scoped to the namespace. When a process in the container tries to mknod or setcap, the kernel checks the mapped host UID, not the namespace UID. Without user namespaces, containers share the host's UID space — if allow userns=0 isn't enabled, an escape from a root-in-container process becomes real root on the host. However, user namespaces add complexity: (1) Mounting filesystems requires --privileged or CAP_SYS_ADMIN in the initial namespace. (2) The subuid and subgid files (in /etc/) define the mapping ranges used by tools like Podman. (3) Not all kernel interfaces are namespace-aware — for example, /proc/sysrq-trigger is global. Docker by default blocks user namespace remapping for networking (since netdev ownership is global), so rootless containers in Docker require extra networking setup (slirp4netns).

jbd2 is the ext4 journaling daemon. High CPU/IO means the journal is a bottleneck — every metadata write goes to the journal first (write-ahead logging). ext4 defaults to data=ordered mode: data blocks are written before metadata is journaled. Under high write concurrency, the journal commits become serialized, and jbd2 flushes the journal to disk at each commit (default 5-second commit interval, or sooner if the journal fills). XFS uses a different design: it allocates log space in a circular buffer, groups multiple metadata updates into a single transaction, and uses delayed logging — dirty metadata is accumulated in memory and committed asynchronously. XFS typically handles concurrent metadata writes better because it batches more aggressively and its log writes use asynchronous I/O when the storage supports it. However, both filesystems will show jbd-like activity under extreme write pressure. Mitigations for ext4: increase the journal size (128MB default), place journal on a separate SSD/NVMe device, use barrier=0 if you accept crash-risk, or switch to data=writeback (metadata-only journaling, risk of data corruption on crash).

This is premature swapping. The kernel's page reclaim logic may swap out inactive anonymous pages even when plenty of file-backed page cache is available. Swappiness (default 60) controls the balance between reclaiming file-backed pages vs anonymous pages — at 60, the kernel is slightly biased to reclaim file-backed pages. If the working set includes many mapped pages (e.g., Java heap, mmap'd files), the kernel may swap them to avoid evicting cached files. The fix: sudo sysctl vm.swappiness=1 (or 0 on kernels < 5.8+ where 0 disables swap entirely). Also check vm.vfs_cache_pressure — lowering it retains dentry/inode caches. Additional diagnostics: sar -S to see swap rates, cat /proc/meminfo | grep -i swap to find what's swapped (use smem or swap from procps-ng). If the application is Java, verify the JVM heap fits in RAM and consider -XX:+UseTransparentHugePages to reduce TLB misses. For databases, disable swap entirely (swapoff -a) or set swappiness=1 and configure vm.overcommit_memory=2 to prevent memory-overcommit-based swapping.

targeted applies SELinux policy only to specific "targeted" daemons (httpd, sshd, named) while leaving other processes unconfined. strict applies policy to all processes, including user sessions — rarely used in production because it breaks almost everything unless the policy is custom-tailored. An "AVC denied" with correct rwx means the file's SELinux context (user:role:type:level) does not match the process's domain. For example, httpd runs in httpd_t domain, but the file is labeled user_home_t (default for files in /home/). Fix: chcon -t httpd_sys_content_t /path/to/file or semanage fcontext -a -t httpd_sys_content_t /path(/.*)? then restorecon -Rv /path/. To troubleshoot: run ausearch -m avc -ts recent | audit2why to decode the denial. audit2allow generates a local policy module to permit the access, but in production, you should instead set the correct context. For debugging temporarily: setenforce 0 (permissive mode) — but never leave it disabled in production.

In the service unit: set [Service] Type=simple with no concurrency control — systemd will start a new instance whenever the timer fires. To prevent overlap, use [Unit] Conflicts=myservice.service combined with [Service] RemainAfterExit=yes — but this is convoluted. The idiomatic systemd solution: in the timer unit, set Unit=myservice.service without OnCalendar only — that doesn't prevent overlap. Instead, configure the service to be Type=oneshot and add [Service] StartLimitIntervalSec=0 with Restart=no, and in the timer, set OnCalendar=*:0/5 with AccuracySec=1s — but this still allows overlap. The correct approach: set [Unit] in the service to StopWhenUnneeded=no and use [Service] ExecStartPre=/usr/bin/flock -n /var/lock/myservice.lock. Better: add [Unit] ConditionPathExists=!/var/lock/myservice.lock and have the script create the lock. The cleanest systemd-native approach: use a [Unit] with BindsTo=myservice.service and After=myservice.service but that's tricky with timers. The practical answer: use flock -n /tmp/myservice.lock /path/to/script in ExecStart — the -n exits immediately if lock is held, and FailureAction= in the timer can handle errors.

top shows aggregate CPU but not per-function breakdown. Use perf top -p <PID> to see the hottest functions in real-time, with symbol names. For JIT-compiled code (Node, JVM, .NET), use perf top -p <PID> -k 0 (to skip kernel symbols) with -g for call-graph. If symbols are missing (stripped binary), install debug symbols or use perf top -f for raw instruction-pointer sampling. Other tools: flamegraph via perf record -F 99 -g -p <PID> -- sleep 30 && perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg. For Python: py-spy or cProfile. For Java: async-profiler. For short bursts, use perf stat -p <PID> -e cycles,instructions,cache-misses,branch-misses -- sleep 10 to see microarchitectural bottlenecks, then drill with perf record and perf annotate. If the binary is compiled with -fno-omit-frame-pointer, call graphs are accurate; without it, perf uses DWARF unwinding (--call-graph dwarf).

You can create a transient systemd timer using systemd-run. The command: systemd-run --user --on-calendar=daily --unit=cleanup /path/to/script.sh — but for 6-hour intervals: systemd-run --on-active=6h --unit=cleanup6h /path/to/script.sh. However, --on-calendar requires proper calendar syntax. For a repeating 6-hour schedule: systemd-run --on-calendar="*:0/360" --unit=cleanup /path/to/script.sh. If you cannot use --on-calendar, use systemd-run --unit=cleanup.service --timer-property=OnUnitActiveSec=6h /path/to/script.sh. This creates a transient timer and service unit in /run/systemd/transient/ that persists until next reboot. For reboot persistence without package installs: write a service unit to /etc/systemd/system/ directly (no systemctl daemon-reload needed if you use systemctl link). If even file write is restricted, use systemd-run --user --scope to create a scope, but it won't survive reboot. The --user instance requires lingering enabled (loginctl enable-linger). The --on-active creates a monotonic timer that runs once — for repeating, use OnUnitActiveSec in the timer properties.

Diagnose: (1) Run mount | grep nfs to list all NFS mounts. (2) Run cat /proc/mounts | grep nfs and check the soft vs hard option — hard mounts block I/O indefinitely if the server is unreachable. (3) Use strace -p <PID_of_ls> to see if it's stuck on stat or getdents on an NFS path. (4) Run nfsstat -m for per-mount statistics. (5) Quick fix for a hanging mount: umount -f -l /path/to/mount (force lazy unmount) — the filesystem is detached immediately, processes get I/O errors and resume. (6) For a soft recovery: mount -o remount,soft,timeo=100,retrans=3,retry=0 /path — soft causes operations to timeout and return errors instead of hanging; timeo (tenths of seconds) sets the retransmission timeout. (7) Long-term: use soft,intr,timeo=600 for NFSv3 or soft,timeo=600 for NFSv4; monitor NFS server health and network latency; use automount (autofs) to unmount idle NFS shares automatically.

Discrepancy means a process holds open file descriptors to deleted files. When a file is deleted while open, du no longer counts it (no directory entry), but df still includes it because the inode is referenced. Find the offenders: lsof | grep "(deleted)" | awk '{print $2, $NF}' to list PID and file path of deleted files. Then check ls -la /proc/<PID>/fd/ | grep deleted for exact sizes. Recovery options: (1) Kill the process holding the file — the kernel releases the space: kill <PID>. (2) If the process cannot be restarted, truncate the file: : > /proc/<PID>/fd/<FD_NUM> — this empties the file content in-place, releasing disk space without interrupting the process. (3) For log files that grow continuously, configure logrotate with copytruncate (copies log then truncates) instead of create (creates new inode) to avoid this scenario. Preventive monitoring: alert on disk_usage > 80% and track df -i for inode exhaustion separately.

The OOM killer uses oom_score = total_rss + total_swap + (total_pages * page_table_factor) / oom_score_adj. The key factor: oom_score_adj (range -1000 to 1000). By default, all processes have oom_score_adj=0. A CPU-intensive process that uses little memory (most CPU time is spent in tight loops on cached/pinned data) has a low oom_score because RSS and swap are low. The OOM killer targets the process with the highest oom_score, which is typically a memory-intensive process (database, cache) that happens to be using lots of RAM. To protect memory-heavy but critical processes, set echo -500 > /proc/<critical_pid>/oom_score_adj (less likely killed). To target the CPU-hungry process: echo 500 > /proc/<cpu_hog_pid>/oom_score_adj which increases its score. But the best fix is addressing the actual problem — either reduce memory pressure or limit the CPU hog. Use systemd-run --scope -p MemoryMax=<limit> -p CPUQuota=<limit> <command> to constrain the process. For production: never rely on OOM killer — set appropriate cgroup limits (memory.max in cgroup v2) per service so the kernel OOMs only within the cgroup, not globally.

Start by proving scope and recent change: top, mpstat, pidstat, perf top, run queue length, and application profiling. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check user/system/iowait/steal time, run queue, hot threads, and recent deployments. Tune the bottleneck, not just instance size.

Start by proving scope and recent change: dmesg, journalctl, /proc/meminfo, smem, ps aux --sort -rss, and cgroup memory stats. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Identify RSS growth, cache behavior, cgroup limits, and OOM scores. Add memory alerts, fix leaks, and protect critical services with sane limits.

Start by proving scope and recent change: df -h, du -xhd1, lsof +L1, journal size, inode usage, and logrotate status. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check both bytes and inodes, find deleted-open files, rotate logs, separate data volumes, and alert before critical filesystems exceed safe thresholds.

Start by proving scope and recent change: curl -w, dig, ss -ti, mtr, tcpdump, MTU checks, and conntrack stats. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Break latency into lookup, connect, TLS, first byte, and transfer. Validate route, MTU, retransmits, and connection reuse before blaming the app.

Start by proving scope and recent change: systemctl status, journalctl -u, systemd-analyze verify, unit environment, and exit codes. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Compare interactive shell versus unit environment, working directory, user, limits, dependencies, and sandboxing. Add explicit env files and meaningful restart policies.

Start by proving scope and recent change: lsof -p, /proc/<pid>/limits, ss -s, systemd limits, and application connection pool metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Find whether descriptors are sockets, files, or leaks. Tune per-service limits, pool sizes, keepalive, and alert on descriptor growth.

Start by proving scope and recent change: ss -s, ephemeral port range, TIME_WAIT counts, conntrack usage, and retry metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Tune connection reuse, keepalive, pool sizes, and ephemeral port range carefully. Prefer application pooling before aggressive kernel changes.

Start by proving scope and recent change: namei -l, stat, ACL checks, SELinux/AppArmor status, and service user identity. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check every parent directory permission, ownership, ACLs, and MAC policy. Use dedicated service users and deployment automation that sets ownership deterministically.

Start by proving scope and recent change: ausearch, audit2why, AppArmor logs, context labels, and policy status. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Confirm MAC denial, fix labels or profiles narrowly, test in permissive/report mode, and avoid blanket disablement.

Start by proving scope and recent change: timedatectl, chronyc tracking, NTP source checks, and drift metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use a reliable time source, monitor offset, avoid manual time jumps, and make alerting sensitive to clock drift.

Start by proving scope and recent change: uptime, top, vmstat, iostat, blocked process stacks, and run queue analysis. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Load includes runnable and uninterruptible tasks. Check IO wait, blocked tasks, storage latency, and locks before buying more CPU.

Start by proving scope and recent change: iostat -xz, pidstat -d, filesystem errors, queue depth, and database IO metrics. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check await, utilization, queue depth, fsync rate, and noisy neighbors. Tune application write patterns and storage class together.

Start by proving scope and recent change: resolvectl, dig +trace, /etc/resolv.conf, packet captures, and resolver logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Check resolver order, search domains, timeout/retry values, negative caching, and application DNS cache behavior.

Start by proving scope and recent change: ps -eo, parent PID inspection, systemd cgroup view, and signal handling tests. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Ensure PID 1 reaps children, services handle SIGTERM, and supervisors do not double-fork unexpectedly. Alert on process count growth.

Start by proving scope and recent change: package changelogs, service restart list, canary host metrics, and maintenance window logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Patch canaries first, know which services restart, drain traffic, snapshot critical hosts, and separate emergency security fixes from routine upgrades.

Start by proving scope and recent change: ss -ltnp, iptables/nft rules, cloud firewall rules, route checks, and packet counters. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Verify listener, local firewall, route, upstream firewall, and return path. Open the narrowest source, destination, and port required.

Start by proving scope and recent change: backup logs, checksum validation, restore test output, and application consistency checks. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Test restores regularly, capture RTO/RPO, validate application-level consistency, encrypt backups, and restrict delete permissions.

Start by proving scope and recent change: journalctl --since, app logs, auth logs, kernel logs, and correlation IDs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Build a timeline by host, service, deployment, and user impact. Centralize logs, preserve evidence, and redact sensitive fields before broad sharing.

Start by proving scope and recent change: serial console, rescue mode, journalctl -b -1, fstab validation, and disk checks. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use console access, boot rescue, validate fstab and kernel args, snapshot before repair, and automate config validation before rollout.

Start by proving scope and recent change: last, auth logs, authorized keys audit, sudoers review, and identity provider logs. Check logs, metrics, health checks, dependency errors, and config drift before changing anything. Recover with the smallest reversible action, then document the root cause. Prevention: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Design for failure domains, rollback, observability, and least privilege first. Validate capacity, limits, network paths, and operational ownership. The practical answer is not one service or command; it is the architecture plus the runbook. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Baseline current behavior, add guardrails in report-only or staged mode where possible, and test the highest-risk paths first. Roll out with logs, alerts, and a rollback plan. Use least privilege, explicit ownership, and automated checks. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Separate build, deploy, validation, and cutover. Use canary or blue/green where possible, keep the old path available until health checks pass, and define rollback before starting. Watch saturation, errors, latency, and user-facing checks. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.

Look at utilization, error rate, latency, queue depth, throttling, quota, retry volume, and recent deployment history. Optimize the bottleneck rather than guessing. Prefer rightsizing, caching, batching, and lifecycle policies before broad rewrites. For this scenario: Use individual identities, centralized SSH certificates or SSO, short-lived access, audited sudo, and automated offboarding.