CI checks out a specific revision, builds and tests it, and publishes an immutable artifact with its revision and digest. Trigger it through the repository integration, such as a webhook. CD receives the tested artifact’s identifier through an upstream-success trigger or an explicit promotion request; it should not rebuild from a moving branch.
Keep deployment permissions separate from ordinary build permissions. Use environment checks and approval gates where required. Record which revision and artifact reached each environment so rollback and audit can refer to a known release.
02
Question 2MediumBuild diagnosis
The app works locally but fails in Jenkins. How would you isolate the difference?
Start with the failed stage and the first useful error in the build log. Confirm the checked-out revision, dependency lockfiles, runtime/tool versions, environment variables, credentials scope and working directory. Compare the agent’s operating system, permissions and network access with the local environment.
Reproduce using the same declared build environment, ideally with a clean workspace or pinned container image. Check whether stale caches hide missing dependencies. Fix the reproducibility gap rather than granting broad permissions or repeatedly rerunning the job without evidence.
03
Question 3HardRelease recovery
A deployment fails after Jenkins has started it. What should recovery check?
Stop further promotion and establish what actually reached the environment. Check rollout state, health signals and the deployed artifact digest. If rollback is appropriate, restore a known-good immutable artifact and verify service recovery. Do not treat a red build result as proof that the deployment made no changes.
Keep failure handling and notifications in explicit Pipeline stages or post actions. Database migrations and external side effects may need a separate recovery plan; rolling back an image does not undo them. Preserve the failure evidence and record the recovery decision.
Jenkins marks a production deployment successful, yet customers cannot complete orders. How do you separate pipeline success from application health, restore service and harden release gates?
The image deploy and Kubernetes rollout stages show green. Replicas are Ready and the build artifacts are signed, but the checkout step returns errors after a database migration and feature-flag change. The pipeline only checked that manifests applied successfully.
Show answer+
Triage: Identify the first bad release and the exact user-journey failure from external synthetic requests, edge 5xx, distributed traces and application logs. Verify the running image digest, environment/namespace, config and secret versions, feature flags, ingress routing, backend service health and database schema compatibility. Check which Jenkins stages assert actual user-visible behavior rather than only command exit codes, and compare the last healthy deployment with the present configuration. Separate image delivery, workload readiness, integration correctness and customer outcome.
Containment: Stop further releases and disable the risky flag or route traffic to the known-good version if it can safely operate with the current database. A database migration may be irreversible or incompatible with an old binary; do not blindly roll back an application without checking forward/backward schema compatibility, data writes and required recovery steps. If canary traffic is available, halt promotion and preserve healthy traffic. Assign an owner to validate recovered transactions, not just pod status, while keeping changes and their provenance recorded.
Prevention: Make build and deploy evidence necessary but insufficient release criteria. After rollout, require authenticated synthetic smoke tests of a real critical workflow, required dependencies and external path, monitored SLO/error burn, and explicit progressive-delivery abort conditions. Use expand/migrate/contract patterns for schema changes, safe feature-flag defaults, artifact digests and environment parity. Distinguish rollback readiness from an approval to perform a potentially unsafe data rollback.
Trade-offs
End-to-end post-deploy tests increase pipeline time and need realistic test accounts; purely infrastructure-focused health probes can pass while the actual checkout contract is broken.
Common mistake
Claiming that Jenkins success, image push and 100% Ready replicas prove the production release works, then immediately rolling back a potentially incompatible schema.
Follow-up
The prior image expects the old database schema, but the migration is already live. What must you verify before shifting traffic back?
Separate deployment completion from release health
Verify artifact, environment and promoted release
Reproduce critical customer transaction
Compare code, flags, configuration and DB schema
Stop promotion; assess safe rollback or feature disable
Validate customer journey and make it a release gate
Check your reasoning
Explain how a green pipeline can coexist with a broken application.
Describe one database migration change that prevents a simple image rollback.
Name at least two externally observed post-deployment acceptance signals.
05
Question 5MediumSecurity
Why shouldn't builds run on the Jenkins controller?
The controller has four executors and most jobs run there because agents are slow to start.
Show answer+
A build on the controller runs with access to the controller's file system, which holds configuration, credentials and other jobs' data. A malicious or careless build script can read or change all of it.
Set the built-in node's executors to zero and run builds on agents, ideally ephemeral ones such as Kubernetes Pods created per build and thrown away afterwards. That also keeps the controller responsive under load.
Trade-offs
Ephemeral agents add start-up time, which caches and pre-pulled images can reduce.
Common mistake
Leaving controller executors on "just for small jobs".
Follow-up
How would you keep agent start-up fast for short jobs?
06
Question 6MediumPipelines
How do you share pipeline logic across 100 repositories without copy-paste?
Each repository has a 300-line Jenkinsfile, and a security fix must be copied to all of them.
Show answer+
Create a shared library in its own repository, with global steps under vars/ such as standardBuild(), and load it with @Library('org-pipeline@v2'). Repositories then call a few high-level steps, and a fix ships by releasing a new library version.
Pin versions so a library change cannot break everyone at once, and test the library with a few pilot repositories before moving the default. Remember that trusted global libraries run outside the Groovy sandbox, so treat changes to them as privileged code.
Trade-offs
A shared library centralises fixes but becomes a dependency every pipeline relies on; version it carefully.
Common mistake
Loading the library from its default branch so every commit to it changes every build immediately.
Follow-up
How would you roll out a breaking library change to 100 repositories?
07
Question 7HardCredentials
A credential appeared in build logs despite withCredentials. How did it leak?
A deploy step prints the API token in plain text in the console log.
Show answer+
Masking replaces the exact secret string in console output. It misses transformed values, such as base64 or URL-encoded copies, or a value printed by set -x in a different form.
Groovy string interpolation is another leak: sh "curl -H 'Authorization: ${TOKEN}'" puts the secret into the generated command, where it can appear in process listings. Use single quotes so the shell expands the environment variable: sh 'curl -H "Authorization: $TOKEN" ...'. Rotate the leaked credential and limit who can read build logs.
Trade-offs
Masking is a safety net, not a control; scripts must still avoid echoing or transforming secrets.
Common mistake
Deleting the build log and not rotating the credential.
Follow-up
How would you scan existing build logs for leaked tokens?
08
Question 8MediumOperations
Jenkins controller restarts lose running builds and hand-made config. How do you make it resilient?
A plugin upgrade crashed the controller; settings had to be rebuilt by hand from screenshots.
Show answer+
Manage configuration with Jenkins Configuration as Code, so controller settings, credentials references and tools come from YAML in Git and can be rebuilt. Back up JENKINS_HOME for history and job state, and pin plugin versions, testing upgrades on a staging controller first.
Pipeline durability settings decide whether running pipelines can resume after a restart; ephemeral agents keep the controller free of build state. With that in place, a broken controller is rebuilt rather than repaired.
Trade-offs
Higher pipeline durability survives restarts but writes more to disk and can slow busy controllers.
Common mistake
Upgrading plugins directly on production without a tested plugin list.
Follow-up
How would you rebuild a controller from scratch in under an hour?