This is related to my previous post:

On modern microarchitectures: can I expect a serious loss of performance between loops which fit on 1 cache line compared to 2 cache lines?

Are there compeling achitectural reasons to really try hard to fit such small loops on 1 cache line instead of 2?

(for instance some "register like" storage for a small number of machine instructions?)

It does not matter how many cache lines your loop occupies (whether 1 or 10), because the CPU reads program memory speculatively on 8-byte boundaries (it always clears the 3 least bits of the address). The system assigns the "Write-Back" (WB) cache type to application pages (via the PAT mechanism), whereas MMIO memory, for instance, is assigned the "Uncacheable" (UC) type, which disables speculative access. Once data has been loaded from RAM into the cache, it exists simply as a contiguous array of bytes and, consequently, has no further impact on performance.

Multithreading, however, is a different matter. Whenever possible, you should strictly pin your thread to a single processor core—in Windows, this is done using the `SetThreadAffinityMask()` function. This is primarily due to the Write-Back caching policy. If a thread executes on core(0) during one quatnum, the scheduler might move it to core(3) for the next quatnum; this causes the cache on the original core(0) to be cleared, forcing the data to be reloaded from RAM into the cache of the new core(3). But if you assign your thread to one specific core right from the start, the scheduler will always dispatch it to that same core, ensuring your data is always waiting for you in the cache.

I appreciate the attempt, but this is not what I am talking about.

I am talking about the machine instruction "window". It is not only a matter of cache memory.

When a loop fits entirely within the Instruction Decode Queue (IDQ):

1. Detection: The hardware monitors backwards branches. If the target of a backwards branch is currently resident in the IDQ and the total loop size does not exceed the IDQ buffer allocation limit, the Loop Stream Detector (LSD) locks the loop into the queue.

2. Clock Gating: The CPU completely disables the entire front end upstream of the IDQ:
a. L1i tag and data arrays are powered down.
b. The Branch Target Buffer (BTB) and Branch Prediction Unit (BPU) are inactive while looping.
c. The Macro-Instruction Translation Engine (MITE/decoders) is clock-gated.
d. Even the Decoded Stream Buffer (DSB / uop cache) is bypassed.

3. Delivery: The IDQ continuously circulates the cached uops directly into the register alias table (RAT) and allocator at the core’s maximum decode/rename width (e.g., 4 to 6+ uops/cycle) without encountering fetch bubbles or decode throughput bottlenecks.

Key Operational Constraints

Strict uop Capacity: It counts unfused/macro-fused uops, not instruction byte counts. Complex instructions (e.g., microcode sequencer transitions via MSROM) will disqualify the loop.

Single Backwards Branch: The loop body must typically end with a single conditional or unconditional backwards branch and contain no calls, returns, or complex nested control flow that cannot be fully buffered.

Hyper-Threading Impact: The IDQ is statically or dynamically split between SMT threads when both hardware threads are active, effectively halving the available uop window for LSD engagement on affected implementations.

The Skylake Erratum (SKL150): On Skylake and Kaby Lake parts, a race condition in the LSD during hyper-threaded execution could cause unpredictable behavior (e.g., spurious execution spikes or register corruption). Intel resolved this by issuing microcode patches that disabled the LSD entirely on those steppings, falling back to the Decoded ICache (DSB). Subsequent core revisions (Ice Lake, Golden Cove) re-architected the mechanism.

---

What this means for bytes of code depends on how the architecture converts instructions to uops -- fewer is better. The IDQ can contain up to 72 uops on some architectures, but only 28 uops on Nehalem. ---

AMD Zen has a much larger and more flexible op cache - no LSD is used. Mainly, just optimize for pipeline filling.

It all depends upon which core is running the code, and which system it is running in.

There is no overall rule that works for all cases. There is no general rule that works "most" of the time. Use cases differ, so outcomes differ.

Some CPUs have different core types, E core and P cores. Maybe tomorrow there will be more types to consider ("AI" optimised cores, or something).

Some cores are very weak. Low cost systems. Narrow internal bus widths.

Some cores are very strong. Server CPUs, high-end "enthusiast" CPUs. Wider internal bus widths.

Some systems run 10's of thousands of threads on hundreds of cores constantly evicting cache lines of the cores when switching threads. Reloading 10 cache lines takes longer than reloading 1 cache line.

Some systems run light loads and a single process gets all the runtime. Cache reload never happens.

The only way to know is to test each option and see which works best in the end use case. Everything else is just guessing. Don't guess, measure.

@bitrake ok. Those decoding cache seems quite big then, just no aggressive LSD on AMD.

I could write the code to use more registers in order to favor out-of-order execution (to ease input/output dependency tracking), but I may miss some macro ops fusions which do expect to 'link' the output/input of a sequence of instructions. Or I am plain wrong?

Performance unrolling, or do you mean something else?

I wouldn't artificially use more registers unless your testing has demonstrated something unexpected - as the registers are just proxies for their rename targets.

SOGs note particular stalls or false-dependencies. Those should be a higher concern than trying to preempt OoO execution.

In the past people would post algorithms and challenge speed/size; but life gets too busy. It would be nice to see such posts in the future. Every problem has sufficient complexity from its dynamics of use (hardware, memory, content) through the potential implementations.

Well, for instance I would interleave the machine instructions for various ALU independent computations instead of having those ALU computations in a sequence.

I wonder how much this interleaving may impact macro op fusion: as far as I know, macro op fusion expect machine instructions to immediatly follow each other (on RISC-V it is explicit in the specs, for instance div/rem)? Unless the CPU does some dependency tracking on a set of machine instructions in some code window and then detect macro op fusion on non sequential machine instructions?

Modern x86 processors have the same requirement of operation and size. For example, XOR EDX,EDX / DIV RCX does not fuse; but XOR RDX,RDX / DIV RCX does -- same for the sign extension variant.

For those fusions to work, the machine instructions must immediately follow each other or you can insert in between some other machine instructions until they don't "intersect" and until they are in some "code window"?

There is no "code window" for these fusions. They much be adjacent. Furthermore, if they split a decode window (16/32 bytes) then maybe fuse doesn't happen. Need to test.

* AMD Zen 5+ is the exception! It requires the two instructions be in the same decode slot (32 bytes), but no flag modifications between. No other processors allow this. Double-pumped decode - a whole cache line at a time - probably opened up time for this.

So Zen5 is able to perform fusion on a decode slot (half-ish cache line) with non-adjacent machine instructions. This is really, really strong. I am surprised intel is not doing that.

For us, it would mean, roughly, unless a adjacent fusion is explicit (like "div/rem RISC-V in the specs"), if we can, we should interleave the machine instructions of independent computations in order to try to fill the ALU units of the pipeline ASAP.

https://chipsandcheese.com/p/amds-zen-6-family-feeding-the-beast

Yep, 1 cache line gets split on 2 decoders. The µops cache is just ridiculously gigantic. So many ALUs... notice 2 mul units and 1 div unit.

I expect 'fusion' to happen at the µop [queue] level, then having 2 macro ops to µop decoders shouldn't get us worry about half-cache line boundaries.

I now wonder how not to stall too much this distribution of the µops over those units.

With our many cores CPUs nowdays, and the CPUs which have literaly to "learn" our programs, OSes should be really careful about their CPU/SMT allocation for program execution quantum, and be almost paranoid for those tasks pushing a bit the CPUs.

I am curious about the load/store optimizations. Because with such a big store queue, if they include the data, we could almost use that as 'special extra registers' sitting between the real registers and the L1 for code which is not data memory intensive.

And they are doing some level of back and forth in CPU design, namely thought 'good ideas' were not in the end for our software base (It seems this did already happen several times across zen generations).

A bit more GHz, but hardly. Maybe the "2nm" is really for the energy and heat.

Well, it is probably time for an ISA cleanup (to remove what did end up as big mistakes like the status register). Maybe APX is the first step towards that.

Patches for ACE v2 and AVX 10.3 are already hitting gcc.

Good performance read on Zen 5 in practice. My history has been to start with code and develop my understanding through changing code. Later I build the thinking about it, but I want to build the intuition about the code up front.

I thought of something: if the fusion does happen in the µops queue, no way we can arrange our macro-ops to secure a fusion: because we don't know the boundaries.

I think I start to see one of the advantages to have some ISA being actually as close as possible to those µops: a good chunk of this complexity is gone.

The rigidity of the ISA is a double-edge sword though. Too rigid and the pipeline is full of holes (inefficiencies) as the processing doesn't align with the ISA. Too flexible and the pipeline is too complex to pivot on changing tasks. The processor attempts to find a happy place in the middle, or have a split personality.

Even graphics processors play that game, but parts of them are biased toward the rigid - as aspects of the processing is known.

The thing is even the "operation" of a some µop could be strongely correlated to toxic legacy from the ISA for performance/implementation simplicity.

Yeah, now I see it: then a RISC-V decoder for zen would not be a very good idea "enough", as you would need to rethink severely this "µop ISA".

That's exactly what is happening with instruction like ENTER or REP MOVS: there is a little program which executes. Parts of the pipeline are consumed by these little programs which can't have interference from the hot path instructions messing things up. That's the rigid personality.

It's not just for legacy - SIMD and system instructions all benefit from this rigidity. Modern processors are tuned such that these instructions interfere as little as possible - the pipeline get right back to hot path instructions.

It's not toxic because such a mode would be required anyhow.

It is a sweet spot, but things like a status flag register are getting in the way of performance at the cost of some significant complexity: should not have been there. And if some µops have infused on that status flag... toxic. This is only an example.

Well, it is what says some CPU designers.

Newer ISA like RISC-V are there to remove that complexity from the way new implementations can achieve good performance.

Because I read some people doing the following hypocrit argument: "status flag is not slow for the price of very little complexity" perfectly understanding that removing that status flag just removes that complexity out of the way on the road to performance, it is actually "saner then better" without any question from this perspective.