I set out to add in-memory sleep obfuscation to Sliver’s Go implant. The plan was simple: port the well-known Ekko technique, wire it behind a --sleep-obfuscation flag, and call it a PR. Oh boy, was I wrong on it being simple.

What followed was a much longer investigation into an interaction between Windows’ SuspendThread API and Go’s runtime scheduler that, as far as I can tell, isn’t documented anywhere. This post is the story, ending in a working design and a blueprint for anyone else who wants to try this on a Go implant.

Beacon established with sleep mask

TL;DR

  • Ekko’s design (SuspendThread the caller + NtContinue-ROP the encrypt/sleep/decrypt on a thread-pool worker) is fundamentally incompatible with Go’s runtime, for reasons that took a lot of dead ends to understand.
  • The root cause is a race between Windows’ SuspendThread and Go’s entersyscall/exitsyscall P-handoff mechanism. It looks like a deadlock inside the Go scheduler waiting for a P from a thread we froze.
  • The fix is a completely different sleep-mask design: run everything on the beacon’s own thread via QueueUserAPC + an alertable wait, with per-peer Windows-thread-pool timers as safety-net watchdogs.
  • Bonus OPSEC: no RWX memory anywhere in the process image at any point in the cycle.

If you’re only here for the code, it lives in implant/sliver/apc/apc.go which should have by now been added to the sliver’s repo PR.

Setup: what sleep obfuscation is and why we want it

Beacons are noisy. In-memory scanners (moneta, PE-sieve, HollowsHunter, and every commercial EDR that ships equivalent capability) hunt for signatures across every process’s committed memory: known-bad strings, distinctive .text bytes, RWX regions, PE headers in unexpected places.

But a beacon at rest - sleeping between check-ins, which is where it spends 99% of its time - is a giant, static target. Same bytes at the same offsets, cycle after cycle. Trivial to fingerprint.

Sleep obfuscation flips the target for that 99%:

  1. Right before entering the sleep window, encrypt the process image in place.
  2. Sleep for the configured interval.
  3. Right after waking up, decrypt and continue normally.

For the 99% of the time you’re scannable, your .text and other sections look like random noise. Different noise every cycle if you rotate the key.

One of the most influential open-source implementation of this is Ekko by Cracked5pider. Its technique, roughly:

  • Suspends the beacon’s own thread and captures its register state.
  • Queues a chain of thread-pool timers, each NtContinue-hijacked to run a specific step (encrypt, sleep, decrypt, restore).

It works beautifully in a C beacon. It kept failing in a Go beacon. That’s where the story starts.

First attempt: naive port

The first cut was mechanical: match Ekko’s C code in Go, using syscall.LazyProc to call the Windows APIs. Credit to scriptchildie for its work on goEkko which inspired me to attempt to follow this through.

Then, understand Sliver’s main beacon loop and how to integrate Ekko/sleep mask in it.

This runs cleanly for a while… then hangs.

It doesn’t crash: it hangs. The process is alive, threads are visible, some are running, but ekko/apc.Sleep() never returns, no check-ins arrive at the C2, and Process Hacker shows a handful of threads in Wait:Suspended state with suspend counts of 1 and some other(s) running/resumed.

Time to hang was random: sometimes cycle 5, sometimes cycle 500. Consistently a hang, never a crash.

Diagnostics that lied to us

I burned a lot of time on ideas that turned out to solve symptoms, not the underlying problem.

  1. Self-healing sweep. Query every peer thread’s ThreadSuspendCount via NtQueryInformationThread at the top of every cycle; drain any residual counts. Doesn’t change our behavior meaningfully: hangs still happen.

  2. LockOSThread the goroutine. Prevents the goroutine from being scheduled onto a different OS thread between our GetCurrentThreadId and the syscalls that follow. Necessary — but not sufficient. Still hangs.

  3. Many more countless ideas which turned into nothing, like spawning “sidecar” processes and threads, implementing channels to signal thread states, re-implementing logic into goroutines, multiple sleeps/resumes for good luck (idk at this point), probably 500+ lines of debug messages and megabytes of logs.

Some of these were real bugs that I fixed on the way to the answer. Others were just a waste of time. A few were probably some wrongly-implemented mess. None was the bug.

The one observation that broke it open

I was staring at a hung beacon in Process Hacker. Frustrated, I right-clicked all of the suspended threads and hit “Resume” — just manually.

The whole beacon unhung. Immediately went back to normal operation. Made a check-in. Ran for another 200 cycles before hanging again.

My manual resume - on some other thread, one that wasn’t even related to whichever thread was in apc/ekko.Sleep - unblocked the process. That’s only possible if the process wasn’t stuck in the kernel at all:

It was stuck in userspace, in the Go runtime, waiting for something that only another running Go thread could provide.

This confirmed the looming hypothesis that I had since 2024 - the first time I attempted to implement sleep mask naively in Sliver - there was a deadlock happening somewhere related to the Go runtime, and I didn’t understand enough about it to know a way forward.

Process Hacker Resume

Root cause: Go’s P-handoff race with SuspendThread

Here’s the Go runtime, in the shortest form that matters:

  • M = an OS thread.
  • G = a goroutine.
  • P = a “processor slot” — a scheduling permit. Every G that wants to execute Go code needs an M and a P.

When Go code makes a syscall - and every LazyProc.Call counts - the runtime executes this sequence:

runtime.entersyscall()   // releases this M's P into an idle pool
                         // so other Ms can use it while we're blocked
<kernel syscall>
runtime.exitsyscall()    // tries to reacquire a P before returning
                         // to Go code

Under normal load this is invisible. Under our sleep mask conditions it’s lethal, and here’s exactly how:

  1. Our loop calls SuspendThread(peerN). entersyscall releases our P into the idle pool, while the operation is at kernel level, thus outside userspace.
  2. Between entersyscall and the kernel actually suspending peerN, some other M picks up our idle P. It’s a race with a very short window, but with dozens of syscalls per cycle we roll the dice a lot.
  3. Our SuspendThread returns. exitsyscall tries to reacquire a P. The pool is empty: our old P is held by whichever M grabbed it.
  4. Now the next iteration of our loop suspends that M. It’s frozen mid-execution, holding our P.
  5. Our exitsyscall can’t find an available P. It falls back to the scheduler’s “wait for someone to hand us a P” path — which relies on sysmon (Go’s runtime monitor thread) periodically checking for stuck Ms and reallocating Ps to them.
  6. sysmon is one of the Ms we suspended. It can’t hand off. Nobody can. Our M is parked forever.

https://github.com/golang/go/blob/c1e0cd12965106310acafbb9bd50ad0830a7283a/src/runtime/cgocall.go#L194

The whole process is running fine at the kernel level. Every thread’s state is what you’d expect. But the Go scheduler is deadlocked in userspace, and the beacon can’t make forward progress.

Manually resuming suspended threads - say, sysmon - unwedges the whole thing. sysmon wakes up, notices Ms parked in exitsyscall, hands out Ps, everything moves. It works because the manual resume came from outside our suspended set.

The same race, second flavor: the resume phase

The story above is what happens going into the sleep when we’re suspending peers. But there’s a second, symmetric race waiting on the way out: when we’re resuming them. Same underlying mechanism, different trigger, and I only found it because a run hung between “resume begin” and “Sleep returning” instead of before “suspend done”.

Setting: .text is decrypted and executable again. Peers are still all suspended. We iterate the peer list calling ResumeThread on each. Every ResumeThread is a LazyProc.Call, which as we established goes through entersyscall/exitsyscall.

Iteration i, resuming peerN:

  1. procResumeThread.Call(peerN) — our M does entersyscall and releases its P.
  2. Kernel runs ResumeThread. peerN’s suspend count drops to 0. peerN starts running.
  3. peerN is a Go M that was frozen mid-syscall (usually mid-exitsyscall from its own prior work). It wakes up still looking for a P to acquire. It looks at the idle pool → sees the P we just released → grabs it.
  4. peerN returns to Go code with our P, and lands in a tight scheduler loop — spinning on a lock, walking work queues, whatever the runtime was doing when we froze it. It holds the P without yielding.
  5. Our ResumeThread syscall returns. Our exitsyscall looks for a P. Empty pool. The Ps that exist are held by peerN (running) and the other still-suspended peers we haven’t resumed yet.
  6. Normally sysmon would preempt peerN’s tight loop with a preemption signal and free the P for us. But sysmon is one of the still-suspended peers — next in the resume queue, but we can’t get there because we’re parked in exitsyscall.

Same permanent deadlock. Different mechanism to get there.

The difference in one sentence

The suspend-phase deadlock is caused by our P being held by a thread we froze. The resume-phase deadlock is caused by our P being held by a thread we just woke up that can’t be preempted (i.e. forced to yield the P so it can be used elsewhere) because sysmon — the only thing that can preempt it — is still frozen from earlier in the same cycle. Same random-but-sometimes-happens outcome.

Building it right: the APC design

Once you understand why Ekko can’t work in Go, the fix isn’t to patch Ekko — it’s to build a different design that doesn’t route the sensitive code through Go’s scheduler at all.

The design that stuck:

beacon goroutine calls apc.Sleep(ms)
  │
  ├─ LockOSThread                 (pin goroutine to this M)
  ├─ VirtualAlloc(RW) outside image, copy trampoline in, set RX
  ├─ Build apcArgs struct with trampoline args 
  ├─ Suspend every image-based peer thread
  │     (per-peer watchdog timer armed BEFORE each SuspendThread)
  ├─ Verify all peers are still suspended (no watchdog fired)
  ├─ QueueUserAPC(tramp, self, &args)
  │
  ├─ NtWaitForSingleObject(processHandle, Alertable=TRUE, timeout=NULL)
  │     kernel dispatches trampoline on THIS thread:
  │       │
  │       ├─ VirtualProtect(image, PAGE_READWRITE)
  │       ├─ SystemFunction032 encrypt     ← RC4, key rotated per cycle
  │       ├─ NtDelayExecution(sleepMs)     ← the actual sleep
  │       ├─ SystemFunction032 decrypt     ← symmetric
  │       ├─ VirtualProtect(.text, PAGE_EXECUTE_READ)
  │       └─ ret
  │
  ├─ (kernel returns STATUS_USER_APC)
  ├─ Resume every peer, close handles
  └─ Return to caller

Three key differences from Ekko:

  1. The encrypt/sleep/decrypt runs on the beacon’s own thread, not on a thread-pool worker.
  2. No context capture, no NtContinue, no ROP. The trampoline is dispatched by the kernel as an APC when the beacon thread enters an alertable wait - a documented, blessed Windows mechanism.
  3. The trampoline is hand-written amd64 machine code in a separate VirtualAlloc‘d region outside the process image. It only calls into system dlls (kernel32, ntdll,..) addresses baked into its apcArgs struct — it never re-enters Go’s runtime.

The P-handoff race can’t reach the trampoline: while the trampoline runs, no Go code is executing on this thread. The kernel is doing the dispatch; the trampoline is doing straight-line machine code with no scheduler awareness.

The watchdog: safety net for both P-handoff races

The APC design still needs peer suspension: with .text marked PAGE_READWRITE during encryption, any other Go thread that fetches an instruction from .text will cause an Access Violation (AV) and crash the process. But the peer-suspend loop AND the peer-resume loop are both Go code making LazyProc.Calls — meaning both can hit the P-handoff race, in their respective flavors described above.

We can’t eliminate the race. What we can do is convert it from a permanent hang into a “skip one cycle and self-recover” event. Same mechanism protects both phases: a per-peer Windows thread-pool timer whose callback forcibly ResumeThreads that peer if the beacon is still stuck when the deadline expires.

The callback must not be a Go callback

During a deadlock - if every Go M is suspended - no Ps are available. A Go callback would deadlock in the exact same way as the caller.

As the watchdogs are our last line of defence against a deadlock, they are dispatched thread-pool worker (not one of our image-based Ms, so not suspended) and calls ResumeThread(hThread) directly, with zero Go runtime involvement. No P needed: this ironically uses the same primitive as Ekko, of using CreateTimerQueueTimer.

The same “no Go code allowed” invariant applies to the trampoline itself, for the same reason. Every function pointer it uses (VirtualProtect, SystemFunction032, NtDelayExecution) is baked into apcArgs at Sleep-call time and dispatched via call [rbx+offset] in machine code — no Go runtime involvement in the encrypted window, ever.

Suspend-phase watchdog

Per peer, in order:

  1. OpenThread → handle to the peer.
  2. CreateTimerQueueTimer with procResumeThread.Addr() as the callback and the handle as lpParameter. Deadline = sleepMs + margin (5–15s, proportional to sleep interval).
  3. SuspendThread(peer).

Order matters. The timer must be armed before the SuspendThread that could deadlock us — the P-handoff race manifests inside exitsyscall after the Suspend syscall, so if we tried to arm all watchdogs at the end of the loop, we’d never get there. The very first Suspend that could hang already has its rescue timer ticking.

Once the loop completes (or the watchdog rescues us out of a deadlock), we do two things before starting encryption:

  • Cancel every timer. Using INVALID_HANDLE_VALUE as the CompletionEvent tells Windows to block until any in-flight callback finishes, so after this returns, each peer’s suspend count is stable — nothing else can race with our check.
  • Verify every peer is actually still suspended. Query each peer’s ThreadSuspendCount via NtQueryInformationThread. If any is 0, its watchdog fired during the suspend loop and the peer is now running.

If any peer isn’t suspended, we abort the cycle: drain-resume everyone, skip this cycle’s obfuscation, return normally. Starting encryption with a peer running would cause an Avcess Violation on that peer’s next instruction fetch and take the whole process down. Losing one cycle of obfuscation beats a crash.

Resume-phase watchdog

After the trampoline returns, .text is decrypted and executable again, but every peer is still frozen. The resume loop is exactly as vulnerable to the P-handoff race as the suspend loop was, just triggered by a different mechanism (freshly-resumed peer grabbing our P instead of frozen peer holding it).

So we do the same thing, symmetrically:

  1. For each peer, CreateTimerQueueTimer targeting procResumeThread.Addr() with a fixed deadline of ~8 seconds. Shorter than the suspend-phase timeout because a well-behaved resume completes in milliseconds — any real deadlock here is caught quickly.
  2. Iterate the peer list calling ResumeThread + drain, then CloseHandle.

Timers are cancelled inline as part of each peer’s cleanup.

If the resume loop deadlocks in exitsyscall, the watchdogs on the still-suspended peers fire and forcibly resume them. Whichever peer’s scheduler unwedges first (often sysmon waking up and immediately doing P-handoff to our stuck M) frees a P for our stuck resume call, and it unblocks.

Cost in the healthy path

When nothing races — which is the vast majority of cycles — every timer gets created, then cancelled before its deadline, and no callback ever dispatches. The overhead is one CreateTimerQueueTimer + one DeleteTimerQueueTimer per peer per cycle.

The rescue path fired at roughly 0.05% of cycles in one of my larger tests (three rescues in 6694 cycles), so the overhead is negligible.

OPSEC hardening: no RWX anywhere in the image

The trampoline lives outside the process image, so scanners might still flag it. To minimize that surface:

  1. The trampoline region is VirtualAlloc‘d as PAGE_READWRITE, populated, then VirtualProtect‘d to PAGE_EXECUTE_READ before dispatch. At no point is it RWX.
  2. When the trampoline returns from its encryption cycle, its last step restores .text to PAGE_EXECUTE_READ — not to PAGE_EXECUTE_READWRITE. Some other sleep masks leave the whole image as RWX between cycles for simplicity; this one doesn’t.

.rdata and .data, as well as the starting PE/DOS header are left as PAGE_READWRITE between cycles rather than being restored to their loader-original protections. That’s a small OPSEC compromise (writable memory where writable memory shouldn’t be) but the primary IOC - RWX anywhere in the image - is eliminated.

Thanks to ethanlacerenza for questioning this and pushing me to go the extra mile on OPSEC.

Unencrypted / check-in: Awake state: three regions, no RWX visible in the image

Encrypted / sleeping: Sleeping state: single flat RW region containing RC4-encrypted image bytes

Wrapping up

This was an absolute journey. What started as (I thought) a quick win in late 2024 ended only close to two years later, after a local infosec conference talk inspired me to give this issue another shot.

While the solution is surely not the most elegant, I’m now confident it works.

Working code is in implant/sliver/apc/apc.go heading into the Sliver PR.

If you’re building sleep obfuscation in Go and want to compare notes, or if you find a hang the watchdogs don’t catch, or even better a more elegant way of making Go go to sleep do reach out!

Thanks to the folks at BishopFox for maintaining Sliver, to C5pider whose Ekko implementation was the reference point even where I ultimately had to depart from its design, and to the infosec community who helped brainstorm ideas and push me to go through with this investigation.