so I figured I'd share this.

I'm developing Multiprocessor support for Mathis using this method.

edit: thanks cod3b453.

I think you meant: luckily 11 does set the lower two bits.

This is an interesting hole, as the MP spec says only INIT and STARTUP IPIs should work. What have you tested this on that shows this behaviour?

Baic: Your code looks very interesting!! It's well worth of inspecting.

A few possible problems arises to my mind though.
If I understand your code correctly, your code simulates INIT IPI- STARTUP IPI for all cores.
Does your code work on cpus with disabled cores? Eg. 3 core cpu with 4th core disabled by the cpu manufacturer.

As Brendan points out here:
http://forum.osdev.org/viewtopic.php?f=1&t=23870&p=193979&hilit=startup+ipi#p193979


regards
Mac2004

Actually, this sends an NMI to all-except-self at ISR 2 - AFAIK this should only affect enabled cores and as it is the BP sending an IPI to all APs, there isn't an issue with simultaneous IPIs.

cod3b453: Thanx for the clarification!

regards
Mac2004

Hi,

First, the code itself.

For NMI, the "vector" part is ignored (as it's not a normal interrupt), and probably should be zero (in the same way Intel recommend that the vector be zero for other IPIs that aren't normal interrupts).

There is no guarantee that the firmware left the local APIC's at 0xFEE00000. You should check the Multi-Processor Specification tables or the ACPI "MADT/APIC" tables to determine where the local APIC actually is (even though it's likely that the firmware did leave the local APIC at it's default address).

Extra notes: If you support old CPUs, then for Pentium there's a bug with the local APIC (see the relevant errata if you're curious), and the workaround is to make sure that every write to a local APIC register is preceded by a read from a local APIC register (any register will do - doesn't need to be the same register you want to write to). For future-proofing, you may also want to check if the BSP supports (and/or is currently using) "x2APIC mode" (and you may want to enable x2APIC mode and use that if the CPU supports it but the BSP isn't already using it).


For the idea; it won't work. The AP CPUs should ignore the NMI (although Intel's manual isn't very clear on this). Ignoring the NMI is what I'd call "best case". The "worst case" would be if the AP CPUs don't ignore the NMI.

When the computer first starts, the firmware does a whole pile of initialisation. This initialisation includes starting AP CPUs, checking BIST, adding them to ACPI/MP tables, configuring things like MTRRs and SMRAM base. For some larger systems (modern "dual/multi socket" systems with Quickpath or Hyper-transport) the firmware may also need to use AP CPUs to configure RAM controllers. Because all this initialisation is messy, I'd expect that most firmware puts APs to protected mode and gives them a valid stack as early as possible; and that when the firmware puts the AP CPUs back into the "wait for SIPI" state the AP CPUs are left in protected mode with SS:ESP set to something.

Now, if NMI worked, the AP CPUs may be in protected mode or real mode (or maybe even long mode). If they were left in protected mode then the IDT probably won't be valid and you get erratic/random failure (they'd hopefully triple fault, but may not, depending on whether "trash" looks like a valid IDT entry or not). If the AP CPUs don't crash and the NMI handler is actually started properly, then the first thing the CPU would do (while starting the NMI handler) is to write some data (e.g. "return CS:EIP and EFLAGS") to the AP CPU's stack. You can't know where the AP CPU's stack might be, and therefore can't know what will be overwritten when this happens - will the AP CPU's stack trash your code or some data you need, or will you be lucky and trash something you don't need? You can't know.

For the normal "INIT-SIPI-SIPI" sequence, the "INIT" is like a reset that restores the CPU to a specific state (e.g. real mode), and no stack is used or needed by the SIPI part. It is immune to the problems you'd get with NMI (if NMI isn't ignored).

Then there's starting "non-CPUs". If an AP CPU fails its BIST (Built In Self Test), then the firmware shouldn't mention the faulty CPU in the MP spec and/or ACPI tables to make sure that the OS doesn't attempt to use the faulty CPU. If your code worked, then it would probably also work to start faulty CPUs that shouldn't be started. Also, for older Pentium 4 systems with hyper-threading, often the user disables hyper-threading in the BIOS (e.g. because hyper-threading makes performance worse for their workload); and you'd be starting those disabled CPUs too. Note: As far as I know, this hyper-threading problem might not effect newer CPUs (Nehalem, Sandy Bridge) as hyper-threading can actually be disabled (rather than only being ignored) for these CPUs.

Finally there's detecting if all APs actually did start correctly. If you know that an AP does exist, then you can attempt to start it and then report an error if it fails to start within a certain amount of time. If you don't know if APs exist or not, then you can't detect when one (or more) fail to start when they should.

For these last 2 reasons; all "AP CPU startup" schemes that broadcast IPIs are bad (regardless of whether it's "INIT-SIPI-SIPI" or anything else). In general, the worst type of bug you can ever have is "works most of the time" (can't reproduce, can't fix).

One last thing! Even if you do the "AP CPU startup" correctly; broadcasting NMI to all CPUs using physical destination mode (e.g. for the purpose of a watchdog timer, profiling or something) is still a seriously bad idea (due to the "invalid CPUs" problem). It can be done reliably, but only if you use "logical destination" mode to ensure that it's only sent to CPUs that are present/running (CPUs that have a non-zero value in their "Logical Destination Register").


Cheers,

Brendan

cod3b453: Thanks. I didn't catch that. The fact that it didn't stop the code from working didn't help either. I'll fix it now.

BCOS: I seriously doubt that this method will subvert the disabled cores that AMD produced. AMD and Intel are hardware vendors: they're perfectly capable of disabling their hardware at the HARDWARE level. Writing to the ICR is how you trigger IPIs in general so reading information in Memory will only lead to writing the ICR anyway. I'm simply cutting to the chase (so to speak).

This works on my ACTUAL computers: Core2Duo E6600 and X58 Core i7 920. as this image shows it works in VMware.



BAiC

Hi,



I didn't say anything about CPUs that are disabled at the hardware level by the manufacturer. I was talking about Pentium 4 (Netburst) systems with hyper-threading, where the end user has disabled hyper-threading (and therefore disabled half of the logical CPUs) using a BIOS option. I also mentioned faulty CPUs that fail their BIST - e.g. CPUs that may have worked fine for years, and then developed a fault. I have seen this happen on a (dual Pentium III) server before - BIOS displayed a "WARNING: CPU1 failed" error message and the OS (Linux) ran perfectly well with only one CPU. It does happen.



You've tested on 2 specific Intel CPUs with 2 specific firmwares. Intel have around 6 micro-architectures that had some sort of multi-CPU, each with about 5 models, each with about 10 steppings (I'll ignore micro-code updates). That's roughly 300 specific CPUs that have some sort of multi-CPU from Intel. You're ignoring the ACPI and MP spec tables so you're probably also doing this on CPUs that don't have any sort of multi-CPU too; so for Intel that probably increases the number of specific CPUs to about 500 (I don't know which CPUs you support - maybe things like Pentium aren't supported anyway). Then there's AMD and VIA (and if you support older CPUs, there's IBM, Cyrix, SiS, Transmeta, etc). For a rough estimate lets call it about 700 specific CPUs from all manufacturers.

Then there's chipsets. Each of these 700 specific CPUs could probably be used with one of about 10 different chipsets. That's a total number of "CPU plus chipset" permutations of about 7000. On top of that, different systems with the exact same CPU and exact same chipset may have completely different firmware. If you estimate that there's 3 different firmware versions for each of them, then you end up with a (very rough) estimate of around 21000 permutations of "CPU, chipset and firmware".

Out of those 21000 permutations, you've tested 2 computers. That's about 0.01% of them. That's so few that it could just be called statistical error, or old fashioned (bad) luck.

I also doubt you've tested it on computers that have CPU/s that fail their BIST, and I know you haven't tested it on "Netburst" systems with hyper-threading disabled in the BIOS.

Now do a google search for "Windows dynamic hardware partitioning" and read a few of the search results. You'll notice that some servers allow the hardware to be split into "partitions" (where a "partition" is some RAM and some CPUs, and has nothing to do with hard drive partitions). The "hardware partitioning" idea is actually very old - mainframes used to do it back in the 1970s (it is relatively new for 80x86 though). The idea is that it allows a physical computer to be split into several logical computers; and allows different instances of an OS (and potentially different versions of different OSs) to be running at the same time on the same physical computer (at the hardware level, without any virtualisation or overhead). Microsoft are introducing a new feature called "dynamic partitioning", where they allow these partitions to be merged while an OS is running (e.g. an OS might have 2 GiB of RAM and 4 CPUs before merging a hardware partition, and 4 GiB of RAM and 8 CPUs after the partition is merged). Can you guarantee that your "NMI startup" will work on systems that use hardware partitioning (and that you won't screw up OSs that are running in other hardware partitions in the same physical computer)?

For the computers that it did work on, where were the AP CPU's stacks? What data got trashed? Did you check?

If you send the NMI and 3 AP CPUs respond, how does your software know that there isn't another 4 AP CPUs that failed to respond for some reason? What if no CPUs respond - is it a single-CPU system or not, and how can your software know?

Let's talk about risk management and "word of mouth" marketing. When something works people don't really talk about it much, but when something fails people tell everyone they know. If 100 people try an OS and it works perfectly for 90 of those people, then those 90 people might tell 90 more people that the OS is good. The other 10 people might tell 10 people each that the OS is a buggy piece of crap, and those 100 people will probably tell another 300 people that they heard the OS is a buggy piece of crap. For an OS that works on 90% of computers, 180 people end up thinking it's good and 410 people end up thinking it's buggy.

If you want to make sure that at least half the people hear that the OS is good (to encourage them to try the OS), then how reliable does the OS need to be? Obviously "works on 90% of computers" is nowhere near good enough. For "works on 94% of computers" you'd end up with 188 people hearing it's good and 180 people hearing it's buggy. What if you want most people to hear the OS is good? Is that even possible?

Given everything above; is it worth spending about 1 hour of time (writing code that does CPU startup correctly) to reduce the risk?

I mean it's not like it's hard to do it properly - you need to find and parse some tables (but you need to find and parse those tables sooner or later anyway, for other reasons) and you need to setup some sort of timer (PIT takes 5 minutes if you haven't set something up already and/or can't use the BIOS "ticks since midnight").

Basically, avoiding an excessive amount of unwanted risk adds up to very little extra work; and while your NMI technique is interesting and inventive; it simply doesn't make any sense to use it in practice.


Cheers,

Brendan

BCOS: one: I consider socketed processors to be equivalent to PCI devices so support for THEM is something else entirely..

two: stay the fuck on topic you ignorant prick you sound like a Web Developer exclaiming support for text browsers in a world of graphical ones. We're talking about MultiCore Processors not Single Core ones. none of the Single Core processors you've mentioned actoually applies to the discussion.

as to the question of what happens if they don't appear, then, well, THEY MUST NOT WORK and therefore should not be used.. it is therefore safer in that respect.

Hi,



For "multi-socket", it's nothing like PCI. Hyper-threading, multi-core, multi-socket SMP and multi-socket ccNUMA all use exactly the same "AP CPU startup". From software's perspective, the difference may be ignored (e.g. an OS can detect things like topology and cache sharing and use that information for various optimisations, but an OS doesn't have to and can treat all of it as "plain SMP" if it wanted to without knowing the difference).



Um, OK. Nice to see members of this forum are able to discuss relevant issues in a professional manner.

May I assume that either the OS uses ACPI tables and/or MP spec tables to detect if multiple CPUs are present (and doesn't do AP CPU startup on single-CPU systems), or that the "multiple CPUs" is a minimum requirement for the OS (and the OS is never intended to work on single-CPU)? The only other alternative that I can think of is that the OS takes neither of these approaches and sends NMIs to CPUs that aren't present in the single-CPU case, but for some reason that you neglected to mention you don't think the single-CPU case is relevant, so I'll have to assume it's one of the former alternatives.



In general; if something should work but doesn't, I'd inform the user/administrator (including providing any relevant information to help them find the cause of the problem); rather then ignoring the problem.



Cheers,

Brendan

Risky and ridiculous generalization. Don't. People behaving like BAiC are virtually everywhere.

The moment I saw this hack I liked it, it reminded me of baldr's hack to get into pm in 45bytes, but it is just a hack, and does not work on my Atom, thus it is unstable and not something to rely apon, even if it does work on 2 PCs there is no garentee it will work on my AMD x2.

BCOS: The pupose of the post was to keep it simple. I never said it would work everywhere. I never said I was looking to support anything.

That doesn't make BCOS' posts any less relevant, though, and it doesn't give you carte blanche to behave like a violated teen.

Your idea is interesting, but as BCOS said it's not something that should be used in practice.

Agreed!

Decorum folks! We will learn more from one another if we keep it respectful.