Session log — 2026-07-10 · post-rewire reboot lessons¶

Working notes captured live during morning validation after overnight rack rewire. These are raw learnings to be folded into Ch 4 (First boot), Ch 15 (Troubleshooting), and the console access chapter (Ch 3) as appropriate.

Post-audit resolutions (2026-07-10, ~11:00)¶

The assumption audit ran below and marked several claims as PROVISIONAL. AFTER that ran, the actual discriminator test executed on ser2net TCP :8000 (FTDI → /dev/ttyUSB0, not :9000-9003 where I'd been probing). The clean output resolved several PROVISIONAL claims:

Root-cause #1 (USB-C cable corrupts data): PROVISIONAL → CONFIRMED. FTDI path ran show network cleanly with full formatted output. No bells, no corruption. USB-C path (:9000) still bells when re-tested. Different cable, different USB converter, different tty — every variable controlled. Cable is the cause.
Root-cause #2 (connect ftd restricted-mode hypothesis): PROVISIONAL → FALSIFIED. The FTDI-path test used connect ftd from FXOS and show network executed cleanly. So connect ftd does not restrict command surface.
New CONFIRMED finding: 1210 mgmt is at 192.168.10.59 on 192.168.10.0/24 (AI subnet) — not the home LAN it was on yesterday. Rack rewire moved it to a different upstream port/VLAN.
New CONFIRMED finding (and personal misassumption tax): ser2net maps /dev/ttyUSB* to ports 8000-8020 and /dev/ttyACM* to ports 9000-9008. I spent significant session time probing 9000-9003 for BOTH paths. Once I read the actual ser2net.yaml config, port 8000 gave clean output immediately.

Ground-truth snapshot with all confirmed state (1210 IP, console commands, OpenBao paths, next-move queue): state-2026-07-10-precompaction.md.

Assumption audit (2026-07-10, live-session)¶

Honesty ledger for this doc. Confirmed claims are backed by direct in-session observation; provisional claims are plausible but not yet verified this session; falsified claims were rewritten with a **Corrected 2026-07-10:** note.

30 claims total
17 confirmed — direct in-session evidence (ping/telnet/pexpect/consolepi-menu output). Trust these.
11 provisional — carry > [!NOTE] PROVISIONAL callouts inline. Pending tests:
1. Run avahi-resolve -n ConsolePi.local on the new Pi to re-verify the hotspot-IP behavior.
2. Capture full show version output on FTD to confirm LSP lsp-rel-20240417-2110 and VDB 392.
3. Capture the actual post-login banner text and test the login: regex against a "Last login: … on ttyAMA0" line.
4. Reference the specific run/log that hit byte 0xbd, or generalize the claim.
5. FTDI-path discriminator: run show network over the FTDI + RJ45 backup console (different cable, different USB converter, different tty) — result tells us cable vs. CLI-mode.
6. Direct FTD console login (bypassing connect ftd) and rerun show network — isolates whether connect ftd attach-mode is the problem.
7. Run show network from a raw nc/telnet client (no pexpect) to rule out terminal-mode desync.
8. Confirm SSH-to-mgmt-IP is clean once the mgmt IP is recovered (validates the "SSH sidesteps byte corruption" premise).
9. Execute configure network ipv4 manual … on FTD 7.6.0 and update syntax if it differs.
10. Verify NetworkManager connection name on the actual Pi (nmcli con show) before running the nmcli con mod command.
11. Capture dmesg on the current setup and confirm cdc_acm: urb submit failed signature is present when the cable is marginal.
2 falsified — rewritten:
1. "Post-reboot session may persist 'already logged in'" — did NOT happen in this session; the fresh telnet DID present fw1210ce login: and required a real auth. Section rewritten to remove the unfounded "sometimes" scenario.
2. "FTDI cable physically attached but not detected by the Pi's USB layer" — actually FALSE: consolepi-menu showed ttyUSB0 [9600 8N1] as a directly-connected device. The correct explanation for the 9001–9003 Device open failure is that ser2net's config for those ports was not bound to /dev/ttyUSB0 (config/restart issue), not that the FTDI failed to enumerate. Section rewritten.

Context — what changed physically¶

ConsolePi was swapped for a new permanent Pi
- Old CPU serial 8b93cccd, MAC b8:27:eb:93:cc:cd
- New CPU serial fa01ccba, MAC b8:27:eb:01:cc:ba
- Same hostname (ConsolePi.local), same ser2net config, same /dev/ttyACM0 → port 9000 mapping
ConsolePi's LAN IP moved from 192.168.1.142 → 192.168.1.121
New USB-C cable installed between ConsolePi and 1210 console port (data line quality untested)
The 1210 was power-cycled as part of the rewire
The 1210's mgmt IP (was 192.168.1.161 at end of yesterday) is now unknown — likely a new DHCP lease
Planned later today: static IPs for both the 1210 mgmt interface and (probably) the ConsolePi — see runbook section below

Ch 3 additions — ConsolePi swap / hardware replacement¶

Nuance: `ser2net` port mapping survives a Pi hardware swap¶

If you replace the ConsolePi's hardware but preserve:

The same OS install (or a clean install with identical ser2net config)
The same USB-C console cable position (so /dev/ttyACM0 still maps to the same target)
The same hostname (ConsolePi.local)

...then everything on the client side (telnet URLs, docs, saved connections) keeps working. The only externally-visible change is the LAN IP (new DHCP lease from the new MAC).

Practical implication: peer readers who bookmark telnet ConsolePi.local 9000 are unaffected by a hardware swap; peer readers who bookmark telnet 192.168.1.142 9000 (specific IP) will silently break.

Gotcha: MAC + CPU serial change after hardware swap¶

If you keep OpenBao / documentation entries that pin the ConsolePi by MAC or CPU serial, they become stale on hardware swap.

Recommended: pin by hostname + service (mDNS _consolepi._tcp), not MAC.

Gotcha: mDNS still reports `hotspot` IP for `ConsolePi.local`¶

Regression re-confirmed on the new hardware — this is a ConsolePi behavior, not a Pi-specific one:

avahi-resolve -n ConsolePi.local
# returns 10.110.0.1  (the hotspot IP, not the LAN IP)

[!NOTE] PROVISIONAL — pending test: run avahi-resolve -n ConsolePi.local on the new Pi in this session and paste actual output. This session went straight to the TXT record via avahi-browse (never invoked avahi-resolve), so the hotspot-IP claim is inherited from the prior session and not re-verified on the new hardware.

Real LAN IP is in the mDNS TXT record:

avahi-browse -rt _consolepi._tcp | grep rem_ip
# rem_ip=192.168.1.121

Add this to console-access.md as a !!! tip "Getting the ConsolePi's real IP" callout. Already documented in consolepi-build-guide/trixie-gotchas.md — cross-link from here.

Ch 4 additions — first-boot flow + reboot behavior¶

Nuance: post-reboot the console lands at FXOS, not FTD¶

Yesterday's session left the console attached to the FTD converged CLI (> prompt). After the overnight power cycle, the same telnet ConsolePi.local 9000 lands at:

fw1210ce login:

...and after login (using admin + the password from OpenBao at infra/webui/fw1210ce-admin), we arrive at:

fw1210ce#

That # prompt is FXOS (Firepower eXtensible OS — the platform layer that runs underneath FTD), not FTD converged CLI. FTD converged is >; FXOS is # with hostname.

How to tell them apart (add as a !!! note in ch 4):

Prompt	CLI	Command that works
`fw1210ce#`	FXOS	`scope`, `show version` (shows FXOS `2.16(0.128)`)
`>`	FTD converged	`show version` (shows FTD `7.6.0 (Build 113)`), `show managers`, `show network`

Nuance: `connect ftd` bridges FXOS → FTD¶

From FXOS #, the command to enter FTD:

fw1210ce# connect ftd
Attaching to FTD ...
> _

Now > prompt = FTD converged CLI. All the commands from yesterday's Ch 4/5 apply.

Practical implication: every "log in via console" step in the build guide should include the connect ftd bridging command, or note that if you land at # you need connect ftd to reach the FTD CLI where the interesting configuration lives.

Nuance: FTD 7.6.0 is running on FXOS 2.16(0.128) underneath¶

Add to bill-of-materials.md (or a new "Runtime layers" callout in ch 4):

Hardware       : Cisco Secure Firewall 1210CE (CSF-1210CE, VID V01)
Platform OS    : FXOS 2.16(0.128)                  ← the wrapper layer
Application    : FTD 7.6.0 (Build 113)             ← the firewall service
LSP version    : lsp-rel-20240417-2110
VDB version    : 392

[!NOTE] PROVISIONAL — pending test: capture and paste the full show version output from FTD to confirm the LSP lsp-rel-20240417-2110 and VDB 392 lines. The session evidence only quoted the "Cisco Secure Firewall 1210CE Threat Defense (86) Version 7.6.0 (Build 113), UUID …" fragment — LSP and VDB lines were not observed in this session.

This is useful context for peers wondering "why does my box show FXOS versioning when I typed show version?"

Gotcha: DHCP lease renews / can change on power cycle¶

If mgmt 1/1 was DHCP (as ours was — 192.168.1.161 yesterday), a reboot may bring back a different address. This one didn't come back at .161 — need to recover the new address by running show network in FTD converged CLI from the console (since we can't hit the FDM UI without knowing the IP).

Recovery flow (add as a runbook to ch 15):

telnet ConsolePi.local 9000 from any LAN host (or the workstation directly)
Log in as admin at fw1210ce login: prompt
If prompt is fw1210ce#, run connect ftd to reach >
Run show network → find management0.IPv4.Address
Update OpenBao at infra/webui/fw1210ce-admin (host field) if the value changed
Update any cached DNS / bookmarks

Ch 15 additions — troubleshooting after a power event¶

Gotcha: post-reboot console session may persist "already logged in"¶

Corrected 2026-07-10: this "sometimes" scenario did NOT happen in the observed session and there is no evidence for it. We thought a persisted FXOS session across ser2net's HANGUP_WHEN_DONE might land the client at fw1210ce# without a login prompt. In practice, the fresh telnet after the reboot presented fw1210ce login: normally (Obs 4) and required a real admin + OpenBao-password authentication (Obs 5) before reaching fw1210ce#.

If a genuine bypass ever gets observed in a future session, re-add this gotcha with the actual transcript. Until then, do not carry it forward — it was an unfounded hypothesis, not a lesson.

Gotcha: `Last login: … on ttyAMA0` welcome message trips prompt-matching¶

FTD's underlying Linux logs "Last login" with the tty name (ttyAMA0 on ARM hardware). If your scripting expects login: as the "please authenticate" prompt, the "Last login:" post-authentication welcome will match it and confuse the flow.

Fix in scripting: anchor login: to end-of-line (login:\s*$) or use fw1210ce login: (with hostname prefix) which only appears at the actual login prompt.

[!NOTE] PROVISIONAL — pending test: capture the actual post-login banner text on FTD 7.6.0 / this box and run the naive login: regex against it. The chronology for this session does not include an observation of a "Last login:" string or a script false-matching on it — the gotcha is plausible from FTD/Linux internals but not evidenced in this run.

Gotcha: raw serial bytes may not be valid UTF-8¶

Some FTD boot output (kernel messages, escape sequences, initialization noise) contains bytes outside valid UTF-8 (0xbd was one we hit). Any script that reads the console via pexpect(encoding='utf-8') without an error handler will crash on decode.

Fix: pexpect.spawn(..., codec_errors='replace') or errors='replace' on decode.

[!NOTE] PROVISIONAL — pending test: point to the specific run/log where 0xbd was observed, or drop the specific byte value and keep the general claim ("raw serial bytes may contain non-UTF-8"). This session's chronology does not include a UTF-8 decode failure — the 0xbd specificity may derive from an earlier session and needs to be traced or generalized before it's a trustworthy lesson.

Gotcha: CLI can enter a state where `show network` returns bells¶

Observed: after connect ftd, running show version succeeded, but a follow-up show network returned show\x07network\x07 — the CLI rejecting the command with bell characters instead of executing it.

Not fully diagnosed. Candidate root causes, in order of suspicion:

New USB-C cable with marginal data lines. The rack rewire replaced the USB-C cable between ConsolePi and 1210. Cheap/marginal USB-C cables often physically fit + enumerate as CDC-ACM but drop or corrupt bytes on the data lines. Symptom would be exactly this: show version (buffered command that completes before corruption hits) works; a follow-up command that requires clean byte transmission fails with bells (invalid characters).

[!NOTE] PROVISIONAL — pending test: FTDI-path discriminator (see below). Session evidence (Obs 15–18) is consistent with the USB-C cable being the ttyACM0 source (removing it made :9000 stop opening) but nothing isolates "the cable corrupts data" from "connect ftd desyncs the terminal." Do not treat byte corruption as established until the FTDI path is run clean.

connect ftd from FXOS may drop into a restricted FTD attach mode — not the full FTD converged CLI. Some commands work, others don't. This is distinct from a direct FTD console login.

[!NOTE] PROVISIONAL — pending test: log in directly at the FTD console (bypass connect ftd) and rerun show network. No direct-FTD login has been attempted this session, so the discriminating test has not been run.

Terminal-mode desync between pexpect's line-buffering and FTD's expected input.

[!NOTE] PROVISIONAL — pending test: run show network from a raw nc/telnet client with no pexpect in the loop. The slow-char-send workaround (Obs 12) not fixing the bells reduces confidence in a simple buffering explanation but does not rule out echo/mode desync.

Discriminator test to run: switch to the FTDI + RJ45 backup console path. Different cable, different USB serial converter (FTDI vs CDC-ACM), different /dev/tty* device, different ser2net port. If the FTDI path is clean, root cause is the USB-C cable (#1). If the FTDI path hits the same wall, root cause is the CLI mode (#2). See ch 3 for the FTDI backup wiring.

Prerequisite for the discriminator test: confirm ser2net is actually serving the FTDI device on one of ports 9001–9003. In this session, all three returned Device open failure: Value or file not found (Obs 16).

Corrected 2026-07-10: we initially assumed this meant the FTDI cable was physically attached but not detected by the Pi's USB layer (i.e., no /dev/ttyUSB* node). That is FALSE. consolepi-menu on the Pi (Obs 19) listed 1. ttyUSB0 [9600 8N1] as a directly-connected device — FTDI IS enumerated as /dev/ttyUSB0. The correct interpretation of the Device open failure on 9001–9003 is that ser2net's config for those ports is not bound to /dev/ttyUSB0 (wrong device path, wrong port assignment, or ser2net needs a restart to re-scan), not that the kernel failed to see the FTDI.

Verify enumeration is working (should show ttyUSB0 present):

lsusb | grep -iE "ftdi|serial|1a86|0403"
ls /dev/ttyUSB*
dmesg | tail -20 | grep -iE "ftdi|usb|serial"

Then inspect ser2net's port-to-device mapping and fix or restart:

grep -E '^(90[0-9]{2}|connection)' /etc/ser2net.yaml   # or /etc/ser2net.conf on older builds
sudo systemctl restart ser2net

Re-probe ports 9001–9003 to see which one is bound to /dev/ttyUSB0. If none of them are, add or fix the mapping in the ser2net config.

Workarounds tested that did NOT fix it:

Slow character-by-character send with 40 ms per char
Ctrl-C + Ctrl-U + Enter reset before the failing command

Alternative that would work regardless of console path: SSH directly to FTD management IP once known — SSH channel is bit-perfect (checksummed transport, retransmits) so it avoids the console-side byte-corruption issue entirely. Chicken-and-egg problem: needs mgmt IP, which is what we're trying to discover.

[!NOTE] PROVISIONAL — pending test: once the mgmt IP is recovered, actually SSH in and rerun show network to confirm the "bit-perfect transport sidesteps the fault" premise. The premise is only useful if root cause #1 (byte corruption on the console wire) is real — which is itself PROVISIONAL until the FTDI discriminator runs.

Ch 3 (Console access) update — USB-C data cable requirement, made explicit¶

Add a !!! warning to the console access chapter:

USB-C data cable, not charge-only. Not every USB-C cable is a data cable. Charge-only cables physically fit and negotiate USB power but have no data lines wired — the target device won't enumerate as CDC-ACM at all. Marginal data cables (bad crimps, thin gauge, missing shielding) may enumerate but drop bytes intermittently — plausible symptoms include a serial session that "works for a moment then hangs," CLI commands echoed with ^G bells, or show commands that partially complete and then fail on the next call. If you're troubleshooting a flaky console over USB-C, swapping the cable to a known-good one is a cheap experiment worth running before deep-diving CLI bugs — but treat "the cable is causing the bells" as a candidate to prove, not a diagnosis (see the Ch 15 candidate-root-causes list, all PROVISIONAL until the FTDI-path discriminator has been run).

Test known-good on your workstation: lsusb | grep -i cisco should show the device; dmesg | tail -20 | grep -i cdc-acm should show clean CDC-ACM enumeration with no error messages.

[!NOTE] PROVISIONAL — pending test: capture dmesg on the current setup (both when the USB-C cable is behaving and, if reproducible, when it's not) and confirm the cdc_acm: urb submit failed signature actually appears under marginal-cable conditions. Not observed in this session — pattern is plausible from the Linux USB stack but not evidenced here.

Static IP runbook (planned for later today)¶

Both the 1210 mgmt interface and the ConsolePi will move to static IPs to resolve the DHCP-re-lease-on-reboot gotcha permanently. Add this section as a Ch 4 (or Ch 15) runbook.

1210 mgmt interface (management0)¶

From the FTD converged CLI (> prompt):

> configure network ipv4 manual 192.168.1.161 255.255.255.0 192.168.1.1

[!NOTE] PROVISIONAL — pending test: this command has not been executed on FTD 7.6.0 in this session (planned for "later today"). Syntax is drawn from Cisco docs. Verify on first execution and update the runbook if the actual FTD 7.6.0 syntax differs (arg order, keyword name, prompt for confirmation, etc.).

Where: - 192.168.1.161 — chosen static IP (matches yesterday's DHCP lease so all existing docs/OpenBao stay valid) - 255.255.255.0 — netmask for the /24 home LAN - 192.168.1.1 — home LAN default gateway

Then set DNS:

> configure network dns servers 192.168.1.3
> configure network dns searchdomains uppernyack.com

Confirm with show network, then update OpenBao entry infra/webui/fw1210ce-admin if the IP differs from what's stored.

ConsolePi (Raspberry Pi 5, if using ConsolePi image)¶

Two ways to lock it in — pick one:

Option A — DHCP reservation on the LAN router. Cleanest. Pin the ConsolePi's MAC (b8:27:eb:01:cc:ba) to a specific IP in the router's DHCP settings. Zero config on the Pi itself; survives OS reinstall as long as MAC is preserved.

Option B — Static on the Pi via dhcpcd.conf or NetworkManager (Bookworm+):

sudo nmcli con mod "Wired connection 1" ipv4.method manual ipv4.addresses 192.168.1.121/24 ipv4.gateway 192.168.1.1 ipv4.dns 192.168.1.3
sudo nmcli con up "Wired connection 1"

[!NOTE] PROVISIONAL — pending test: verify the actual NetworkManager connection name on this Pi with nmcli con show before running the modify. "Wired connection 1" is the NM default but the ConsolePi image may ship with a custom name, or may not be on NetworkManager at all (dhcpcd is equally likely on ConsolePi Trixie). Also not yet executed.

Chosen static: 192.168.1.121 (matches current lease so no re-bookmarking needed).

Recommendation: Option A (DHCP reservation) — simpler + more resilient.

Documentation updates queued from this session¶

[ ] Ch 3 (Console access) — add "ConsolePi hardware swap" section + mDNS TXT record IP discovery cross-ref
[ ] Ch 4 (First boot) — add FXOS-vs-FTD prompt table + connect ftd bridging note + runtime-layer BOM detail
[ ] Ch 15 (Troubleshooting) — add "post-reboot recovery flow" + the three scripting gotchas (Last-login trap / UTF-8 decode / show-network bell state)
[ ] Ch 2 (BOM) — add a "if you replace the ConsolePi hardware" reminder
[ ] OpenBao entry infra/webui/fw1210ce-admin — update host field once new mgmt IP is recovered

Later same day (~12:00 – 12:45) · autonomous factory-reset attempt lessons¶

Fabian delegated an autonomous "reset + first-boot capture" run so he could stay hands-off. It failed for reasons entirely worth capturing, then a methodical enumeration on the live box surfaced the correct remote-reset path. Fed into new Ch 3.5.

Root-cause of the failed autonomous run¶

configure factory-default does not exist on FTD 7.6.0-113 for the 1200 series. Enumerated live: configure ? from FTD > lists 30+ submodes (manager, network, password, ...) — no factory-default. The command was bell-rejected by the FTD (\x07) with no visible output. The old troubleshooting doc pointed at it verbatim — that was the seed of the failure.
Cascade: script sent the command → FTD bell-rejected → no confirm prompt → 15s regex timeout → script proceeded to a wait_for(reboot signal) phase for 17 minutes with nothing to wait for. Watchdog eventually fired because the parent process exited on its own timeout. 17 minutes of silence looked identical to "in progress."
Correction: the script's Phase 2 needs a positive-affirmation check ("did the FTD actually accept the reset command?") before entering the wait-for-reboot phase. Silence at that layer must be treated as failure, not as "reset in flight."
CONFIRMED gotcha: pexpect + line-buffered log file will hide a bell-rejection from post-hoc log analysis, because the bell character travels alone with no \n and never triggers a flush until the next full-line write. Rerun the transcript through cat -A to see the bell.

Live-enumerated reset surface on 1210CE / 7.6.0-113¶

Direct ? walk from FTD > → FXOS → all reachable scopes:

FTD > prompt has NO factory-reset command. Only partial-reset paths (configure manager delete, configure network reset, configure password admin) which don't re-arm the setup wizard.
FXOS root (fw1210ce#) has scopes: chassis, eth-uplink, fabric-interconnect, firmware, monitoring, org, packet-capture, security, server, ssa, system. No erase command at root.
scope system on this build: acknowledge, activate, create, delete, enter, scope, set, show — no erase configuration. That command doesn't exist on 1200-series FXOS at this build.
scope security: password/session mgmt only — no reset.
scope security-services: invalid on 1200 series. Security Services Application lives at scope ssa instead (a rename from the 1000/2100 pattern that older docs still cite).
scope firmware → scope auto-install (fw1210ce /firmware/auto-install#) — this is where the reset lives. Commands: acknowledge, cancel, install, show. install security-pack version <ver> force is the Cisco-recommended remote factory reset.
show version inside scope system confirmed the build: Package-Vers 7.6.0-113, Platform-Vers 2.16(0.128), Rommon-Vers 1.1.09.

Serial console default lands in FXOS, not FTD. FTD > prompt is downstream, entered via connect ftd from FXOS.
exit at the FTD > prompt does NOT log you out — it returns you upstream to FXOS Service Manager (fw1210ce#). Old memory said "exit = logout," wrong on this platform.
connect fxos from FTD > prints "You came from FXOS Service Manager. Please enter 'exit' to go back." and does not move you. It's essentially a no-op from FTD. Older troubleshooting notes cite connect fxos as the way to reach FXOS — that's only correct from a non-FTD entry point.

Cisco's recommendation as it actually reads on this hardware¶

install security-pack reimage (via scope firmware/auto-install) is the Cisco-recommended remote factory reset for the 1000/2100/1200 series. Wipes FXOS + FTD in sync so the two layers can't drift.
erase configuration is Cisco-flagged as risky (may leave FXOS mismatched with FTD, especially in HA). On 7.6.0-113 for 1200 series, the command isn't present anyway — the surface pushes you to the reimage path.
ROMMON reset is the last-resort recovery for a locked-out box. Requires a physical boot-interrupt at the console. Not remote, not lights-out.

Result: Ch 3.5 authored around the install security-pack path with a one-time install-package staging on ConsolePi (/srv/firmware-cache/). No shims, no fallbacks — one recommended path documented once.

Ser2net long-session behavior¶

Connection closed by foreign host appeared when the pexpect script sat idle 17+ minutes without input. Ser2net drops idle telnet sessions rather than holding them open indefinitely. Fresh reconnect worked cleanly (banner served + FTD prompt available). Not a bug — expected behavior. Long-running scripts should be structured to reconnect if the session drops.
Ctrl-U (line kill) clears the FTD's input line buffer when it's holding stale characters from a prior partial command (e.g., an aborted abc? that never got Enter). Sending \r alone against a non-empty buffer echoes back nothing useful — the FTD is waiting for line terminator. Ctrl-U first, then \r is the safe wake pattern.
Picocom via consolepi-menu option 1 locks /dev/ttyUSB0 exclusive. When someone's picocom session is live, ser2net cannot open the tty and telnet-to-:8000 clients get "Device open failure: Object was already in use." Only one console consumer at a time on FTDI.

Documentation deltas from this half-day¶

✅ New Ch 3.5 (remote-factory-reset.md) — canonical remote reset procedure with rationale table and one-time staging
✅ troubleshooting.md: "FW in a weird state" nuclear-option command replaced with pointer to Ch 3.5 + explicit stale-command warning
✅ troubleshooting.md: "Console shows nothing after power on" section rewritten around FTDI + RJ45 primary path
✅ mkdocs.yml: Ch 3.5 added to nav between Console access and First boot
[ ] Ch 4 (First boot) — TBD after actual reset execution captures the real wizard flow

Install-package staging session (~13:00) — naming convention nuances¶

Fabian navigated Cisco.com Downloads to fetch the reset install package. Every step surfaced something my Ch 3.5 draft got slightly wrong from extrapolating 1000-series conventions. All corrected in the chapter, but the pattern is worth capturing.

Filename convention change: 1000/2100 → 1200 series¶

1000/2100 series (old Firepower + FP2100): cisco-ftd-fp1k.<version>.SPA — dot-separated, .SPA extension (signed application archive).
1200 series (1210CE / 1210CP / 1220CX — this hardware): Cisco_Secure_FW_TD_1200-<version>-<build>.sh.REL.tar — underscore-heavy, .sh.REL.tar compound extension.
Cisco.com's download page explicitly says "Do not untar" — the .tar file itself is the install package; the extractor lives inside the FXOS installer, not on the operator's workstation.
Every guide I've seen (community, Cisco's own reimage doc) references the .SPA name because they were written for the 1000-series. That's technically correct for their scope; it's just wrong for the 1200 series. Anyone who copy-pastes a .SPA reference and hits a 1200-series box will be off in the weeds.

Download portal path — canonical drilldown¶

Recorded so the guide can point readers at the exact clicks:

software.cisco.com
  → Downloads Home
  → Security
  → Firewalls
  → Next-Generation Firewalls (NGFW)          ← not "Adaptive Security Appliances" (ASA)
  → Secure Firewall 1200 Series
  → Secure Firewall 1210CE                     ← model-specific page
  → Firepower Threat Defense (FTD) Software    ← not "Firepower Coverage" (that's LSP/VDB)

The sibling categories worth calling out so nobody clicks the wrong one:

ASA Device Manager / ASA Software — legacy platform, not FTD. Wrong.
Firepower Coverage and Content Updates — LSP/VDB refresh, not the OS. Wrong.
Firewall Migration Tool (FMT) — for migrating from other platforms. Wrong.

Per-release download page — two files, only one is right¶

At the release-selector page for a given version, Cisco presents two files:

File	Size	Purpose	Pick?
`Cisco_Secure_FW_TD_1200_Hotfix_CC-<ver>.<hf>-<build>.sh.REL.tar`	~220 MB	Common Criteria certified hotfix — regulated deployments (federal, defense, some healthcare)	❌
`Cisco_Secure_FW_TD_1200-<ver>-<build>.sh.REL.tar`	~970 MB	Install & upgrade package	✅

Take the larger one. The CC hotfix is not a reset image and doesn't map onto the install security-pack FXOS flow.

Actual sizes on 1200-series vs my earlier estimate¶

I estimated ~1.5-2 GB for the install package (extrapolated from 1000-series). The actual 1200-series install package is ~970 MB (Cisco_Secure_FW_TD_1200-7.6.4-69.sh.REL.tar is 1,016,965,120 bytes).
Ch 3.5 was updated to say ~970 MB after the correction.

Version-string nuance¶

Cisco.com's release picker shows the version as 7.6.4 (no -69 suffix — that's a build number the picker hides).
The install command on FXOS takes version <ver>-<build> explicitly — install security-pack version 7.6.4-69 force, not install security-pack version 7.6.4 force. Off-by-one on this and the installer will refuse or grab the wrong file.
The FTD's own show version reports Package-Vers: 7.6.4-69 in the same <major>.<minor>.<patch>-<build> format. So there IS a single canonical version string across UI + CLI, but the portal's release picker suppresses the -<build> part in its dropdown. Grab the full string from the filename or the file-details block, not from the dropdown.

Downstream file — what got staged in this session¶

Cisco download page: 7.6.4 (dropdown), install file Cisco_Secure_FW_TD_1200-7.6.4-69.sh.REL.tar, release date 2026-01-07, size 969.85 MB per Cisco / 969.85 MiB verified
Staged: /srv/firmware-cache/Cisco_Secure_FW_TD_1200-7.6.4-69.sh.REL.tar on ConsolePi (192.168.40.5)
SHA256: 469eb3e389f2fa14c57799d87e8290d389e0a53e81103c70b4a25d6fc12636b3 — verified matching on both workstation and ConsolePi after SCP transfer
Version delta: box is currently on 7.6.0-113; reset will land it on 7.6.4-69 (staying on the 7.6 train, moving to current release patch)

Delta into Ch 3.5¶

Filename in the guide corrected from .SPA → .sh.REL.tar throughout
Version reference throughout the chapter changed from 7.6.0-113 to 7.6.4-69
New !!! warning callout on the 1000-vs-1200 naming distinction so readers don't get burned by copy-pasted .SPA from older Cisco Community posts
New table for the two-file choice on the release page
New !!! info callout on verifying install security-pack argument syntax with ? at execution time, since command-flag drift between FP1K and 1200 platforms hasn't yet been verified live

FXOS syntax verification session (~13:20) — pre-execution nuances¶

Ran ? help queries on the actual FXOS scopes before committing to a reset that touches 45+ minutes of unattended time. Captured to captures/fxos-syntax-verify-2026-07-10.log. Three surprises:

1. `install security-pack version <ver> ?` executes the command¶

On FXOS 2.16(0.128) shipping with FTD 7.6.0-113: after ? shows completions, pressing Enter on the same line executes the command with the args parsed so far. ? is not a pure help-only lookup at the version <ver> level.

Safe if the package isn't downloaded yet: FXOS returns Invalid software pack. Please contact technical support for help and does nothing further. That's how the verification session found this — the query was safe by accident, not by design.
Not safe once the package IS staged: at that point, the same accidental Enter would kick off a real install with default args (no force, no verify-only) — probably harmless in a same-version reimage, but not what you want if you're just poking.
Use verify-only for true dry-runs: install security-pack version 7.6.4-69 verify-only is the documented no-effect probe.

Captured to Ch 3.5 as a !!! danger callout so this doesn't ambush anyone building the guide's flow.

2. FXOS SCP URL syntax — no colon between host and path¶

download image scp:? prints scp:[//[username@]server][/path]. Path immediately follows the server. My Ch 3.5 draft had scp://pi@192.168.40.5:/srv/firmware-cache/... with a spurious colon before /srv/. Corrected to scp://pi@192.168.40.5/srv/firmware-cache/....

Other transports available on this build: ftp: http: https: sftp: tftp: usbA:. All follow the same URL-style pattern.

3. Verified `install security-pack version <ver>` completions¶

Enumerated live on the box (install security-pack version 7.6.4-69 ?) — the four available flags on this build:

Option	Meaning
`<CR>`	Interactive install with default confirmations
`force`	Skip "already installed" guard — required for same-version reimage
`starttime`	Schedule delayed upgrade (`YYYY-MM-DDTHH:MM`) — not used for reset
`verify-only`	Compatibility + package integrity check without installing — safe dry-run

For a factory reset: install security-pack version 7.6.4-69 force (or plain force with a same-version reinstall).

FXOS scope-prompt regex — script gotcha¶

When FXOS is inside a scope (e.g., scope system from an earlier session), the prompt is fw1210ce /system #, not fw1210ce#. My initial pexpect regex only matched the root form and hit spurious timeouts. Correct regex for any FXOS prompt on this box:

fw1210ce(\s+/[a-zA-Z0-9/_-]+)?\s*#\s*$

Also — the FTD's --More-- pager and the FXOS prompt do not clear each other. Before authenticating on a fresh telnet, send q (escape pager) + Ctrl-U (clear input line) + \r (fresh prompt). Otherwise a stale --More-- from a prior session eats the login flow.

Reset architecture pivot (~13:45) — from Candidate 2 → Candidate 3, adversarially verified¶

Ran a 5-agent workflow (2 lens analyses × 2 adversarial skeptics × 1 synthesizer, workflow.name=best-reset-1210ce) to pressure-test my earlier "download+install 7.6.4-69" recommendation as SE-facing best practice. Both skeptics refuted it. Both also refuted the "same-version pre-staged only" alternative as a published artifact. Consensus verdict: Candidate 3 — two-chapter runbook.

The refutations that mattered¶

Field-disaster skeptic — Candidate 1 (pre-staged reset only) fails as a published SE playbook:

CDO minimum version rejection: CDO enforces a minimum FTD floor Cisco advances every few quarters. A 1210CE that sat in inventory can pre-stage below the current floor. Junior SE follows the guide, box lands on factory version, CDO refuses onboarding, customer watches.
Enterprise "no unpatched device" policies: post-Salt Typhoon / Volt Typhoon era, most large enterprises (banks, healthcare, federal) have a hard rule that no device with known unpatched CVEs may touch even the management VLAN. Factory pre-stage by definition carries every CVE fixed since RTM.
FMC version-skew rejection: 7.7.x-specific policy elements (newer Snort 3 rules, TLS decryption features, ZTNA) can't deploy against a 7.6.0 reset box even if the compat window is technically satisfied.
Manufacturing drift: different lots ship different pre-stages. The guide can't tell readers what version their box will land on — the guide is deterministic per-unit but non-deterministic per-reader.
"Guide as SE playbook, not reference doc": Cisco can afford to split Reimage from Upgrade in reference docs because SEs cross-reference. An action-oriented runbook on a customer Webex share with a 90-min change window can't punt to "go read another Cisco doc."

Security-posture skeptic — Candidate 1 as a terminal state is professionally hazardous:

PCI-DSS 4.0 §6.3.3 requires critical security patches within one month of release.
NIST 800-53 SI-2 (Flaw Remediation) requires timely installation of security-relevant updates.
HIPAA §164.308(a)(5)(ii)(B) — protection from malicious software including patch management.
FedRAMP Moderate inherits SI-2 with enhancement SI-2(2).
An auditor walking a change ticket that reads "reset per Cisco SE guide, handed off" against an outdated baseline will write a finding — and the SE is holding the pen on that finding.
Junior SEs (the guide's stated audience) treat the published terminal state as the correct terminal state. If the terminal state is factory version, unpatched boxes ship to production.

The structural insight that resolved it¶

Candidate 3's Step 1 IS Candidate 1 verbatim. Every architectural win the Cisco-design lens praised (deterministic, atomic FXOS/FTD resync, air-gap-capable, mirrors Cisco's Reimage/Upgrade doc split, one canonical command) is preserved. Candidate 3 only adds a gated Step 2 chapter that closes the patch-posture gap by construction. There is no lose-column on the pivot from Candidate 1 to Candidate 3 that isn't dissolvable by author discipline (a decision-gate callout at the top of Ch 3.6, an explicit STOP banner at the end of Ch 3.5).

What actually got published¶

Ch 3.5 rewritten around Candidate 1 — same-version reimage from the factory pre-staged package, zero external deps, one command
Ch 3.6 written new — production-readiness upgrade, mandatory-for-production decision gate, download + install flow, air-gap usbA: variant, change-record hygiene for regulated fleets
Ch 3.5 ends with a STOP banner pointing at Ch 3.6's decision gate
Ch 3.6 opens with the decision gate, valid-exception list, and change-record language
Ch 15 troubleshooting updated to route through the two-chapter pattern

Why this matches the guide's deployment-matrix arc¶

The guide walks readers through Layer 0 (console) → Layer 1 (FDM standalone) → Layer 2 (SCC hybrid) → Layer 3 (cdFMC). Every "return to zero" between those layers uses Ch 3.5 only — fast, no download, deterministic. The one-time production handoff at the end of the arc runs Ch 3.5 + Ch 3.6 as a pair. Reset-only and reset+upgrade are the two operations the arc actually needs; giving each its own chapter matches how readers use the guide.

Meta-lesson for me¶

I flip-flopped twice on this decision (Candidate 2 initially, then Candidate 1 after Fabian pushed on lights-out robustness, then Candidate 3 after the workflow). The flip-flops all stemmed from optimizing for one property in isolation rather than treating "published SE playbook" as its own set of constraints. Ultracode's adversarial-verify pattern surfaced the trap — a single lens (either "get to current patch" OR "no external deps") can look decisive until skeptics run at it. Both lenses need to fit the audience.

Live reset execution (~14:33-15:14) — semantics I confabulated, semantics that are real¶

Fired the real reset. Two big semantic corrections and a wizard flow that's completely different from pre-7.6 platforms.

1. `install security-pack version <same> force` is a silent no-op on 1200-series 7.6.x¶

My earlier Ch 3.5 draft (based on the workflow's Candidate 1 verdict) said install security-pack version <pre-staged> force was the canonical reset. It isn't. Live-observed on the box:

firepower# scope firmware
firepower /firmware# scope auto-install
firepower /firmware/auto-install# install security-pack version 7.6.0-113 force

The system is currently installed with security software package 7.6.0-113, which has:
   - The platform version: 2.16.0.128
   - The CSP (ftd) version: 7.6.0.113

Do you want to proceed ? (yes/no): yes
Do you want to proceed? (yes/no): yes

Triggered the install of software package version 7.6.0-113
Force option: true
Install started. This will take several minutes.

30 minutes of silence later, show detail at /firmware/auto-install# revealed:

Firmware Auto-Install:
    Package-Vers:              7.6.0-113
    Oper State:                Scheduled
    Upgrade State:             Update Software Pack Completed
    Upgrade Status:            upgraded
    Firmware Upgrade Message:  up-to-date          ← the truth

force overrides cross-version compatibility warnings, not the same-version guard. The Force option: true line in the output is FXOS reporting your input; it doesn't mean anything for the same-version case.

Practical consequence: the "reset via pre-staged factory package" architecture that the workflow's Cisco-design lens praised and the SE-ops lens accepted-under-Candidate-3 doesn't work as a reset on this hardware unless pre-staged ≠ running. For factory-fresh 1210CE units where pre-staged = running (the common case), the operator MUST stage a different version before reset will actually reimage.

2. Cross-version install DOES reimage — with a completely different output signature¶

Same command with a different target version:

firepower /firmware/auto-install# install security-pack version 7.6.4-69

The system is currently installed with security software package 7.6.0-113, which has:
   - The platform version: 2.16.0.128
   - The CSP (ftd) version: 7.6.0.113
If you proceed with the upgrade 7.6.4-69, it will do the following:
   - upgrade to the new platform version 2.16.1.147
   - reimage the system from CSP ftd version 7.6.0.113 to the CSP ftd version 7.6.4.69
During the upgrade, the system will be reboot

During the upgrade, the system will be reboot — that phrase is only emitted on the cross-version code path. If you don't see it after two yes confirmations, the reset didn't fire.

3. New UUID as reimage evidence¶

Pre-reset FTD UUID: dc0bef0a-66b5-11f1-9951-911e3bad6766. Post-reset: 3c2bdc6c-7c8f-11f1-9ef7-9d09498b33b8. Different. A same-version-force pass would have kept the UUID (further evidence it doesn't actually reimage). Cross-version install regenerates the UUID — Ch 4's verification checklist uses this as one of the three "did the reset actually run" checks.

4. Empirical timing on the reference reset (7.6.0-113 → 7.6.4-69, 1210CE, 2026-07-10)¶

Milestone	Wall-clock	Δ from install fire
`install security-pack version 7.6.4-69` fired + `yes` × 2	14:33:33	0
FTD shutdown sequence starts (`INIT: Completed /etc/rc6.d/K00ftd.sh stop`)	14:34:31	+57s
Mgmt IP dropped (192.168.10.59 → 100% packet loss)	~14:34:35	+62s
FXOS bundle-wipe fault emitted (`Bundle version in firmware package is empty, need to re-install`)	14:37:35	+4m 2s
First `firepower login:` prompt reappears on console	14:50:55	+17m 21s
FTD wizard `Successfully performed firstboot initial configuration steps`	15:13:13	+39m 40s
At FTD `>` prompt with `show version` returning target	~15:14	+40m 30s

My earlier "30-45 min" estimate was WebSearch summarizer output plus training-data extrapolation, not a directly-verified number for this SKU. Actual = ~17-18 min reimage + ~5 min wizard, total ~22-25 min end-to-end.

5. FTD 7.6.4 wizard flow — minimal, and different from what my script assumed¶

The FTD setup wizard on 1200-series 7.6.4 doesn't run automatically at first boot. Sequence:

FXOS-side first login — firepower login: admin / Password: Admin123 — forces password change on next line.
FXOS at firepower# prompt — FTD wizard has NOT yet run.
connect ftd — this is what triggers the FTD first-boot wizard.

The wizard itself (7.6.4) is much shorter than pre-7.6 flows. Live-captured prompts:

Please enter 'YES' or press <ENTER> to AGREE to the EULA:
Configure IPv4 via DHCP or manually? (dhcp/manual) [manual]:
Configure IPv6 via DHCP, router, or manually? (dhcp/router/manual) [dhcp]:
No DNS servers specified to configure.
No domain name specified to configure.
No hostname name specified to configure.
Setting DHCP for IPv4: management0
Manage the device locally? (yes/no) [yes]:
Configuring firewall mode to routed
Successfully performed firstboot initial configuration steps...

That's the whole wizard on 7.6.4. It does NOT prompt for hostname, DNS servers, search domains, proxy, or firewall mode — those default to sensible values (hostname firepower, Cisco/OpenDNS resolvers, no proxy, routed mode). Custom values go through FDM UI in Ch 6 or the configure network FTD CLI commands.

My earlier reset script had wizard patterns for fully qualified hostname, DNS servers, search domains, proxy hostname, and firewall mode — none of those prompts are ever emitted on this build. Script's pattern-matching worked by coincidence (matched adjacent text) and got lucky with defaults.

6. Ch 3.6 collapsed into Ch 3.5 after this run¶

The workflow's Candidate 3 architecture (two chapters — reset + upgrade separately) was based on the assumption that reset and upgrade are semantically distinct operations. On 1200-series 7.6.x, they're the SAME FXOS command with different version targets. Two chapters was documenting an architectural split that doesn't exist on this hardware. Collapsed:

Ch 3.5 (rewritten) — the reset mechanism as it actually works: cross-version install. Includes the package-staging sub-section (previously Ch 3.6) as optional, only needed if the operator doesn't already have a suitable target on flash.
Ch 3.6 — DELETED. mkdocs.yml nav updated, troubleshooting.md updated to point at only Ch 3.5.

The workflow's residual value: the decision gate language (production-readiness) and the change-record hygiene section moved into Ch 3.5's tail. The two-chapter split itself was a false architecture given the hardware's actual behavior.

7. Watchdog pattern that kept failing¶

Every wrapper watchdog I set up followed this pattern:

until ! kill -0 $PID 2>/dev/null; do sleep 30; done
echo "SCRIPT-EXITED $(date -Iseconds)"
tail -80 $LOG

Some watchdogs completed with exit 0 (log intact when tail ran). Two completed with exit 1 because I renamed the log file mid-flight and tail couldn't find it. The reliable pattern going forward: launch the actual pexpect script under Bash run_in_background: true — the harness watches the script's PID directly, notifies me on exit with the script's stdout, no wrapper needed. Post-run redaction happens inside the pexpect script itself so the log is publication-safe when the task-notification fires.