I am designing my own 8086 assembler for my hobby-os.
First i thought to search the web for any hits (google)
I was surprised to only find one single < 8k assembler.
(Which runs under DOS and is written in its own language)
I thought there would be many, but only one?
Maybe the complex instruction encoding scares most people away?
Anyway, here is the one i found...
http://code.google.com/p/x86-s/
I think its pretty cool, minus a few small bugs and syntax weirdness.
So, do you know of any self-assembling assemblers in < 8k ?
If so, i would like to study them.
Writing an assembler is a lot of work and i could save a lot of time
if i could find something usable and tweak it a bit, rather than from scratch.

I do not know if they meet the conditions you require, however you will find a list of sources here
Maybe you'll find something useful.

There is always tiny assembler which was part of the tiny c project in the 1970s. I don't know the home page but you can search online "tiny assembler"

i liked the comment style

Here's what I found so far|

http://code.google.com/p/pe-asm/

http://code.google.com/p/comasm/ ---> You should definitely check this one out.

Hmm, this page says there are no downloads available?
http://code.google.com/p/comasm/
It also says my browser is out of date.
Maybe you could leave link to download file?

Yeah I'm sorry about that. It seems like theere has not been any submitted project files. Sorry

I found one more, although written in C, and somewhat larger in exe.
This one is table driven, MASM syntax, with fewer bugs.
http://www.bbs.motion-bg.com/index.php?file=343
See: GASMSRC\GASM.C (2000+ lines of core, all in one file)
I was planning to prototype my assembler in C anyway.
And MASM syntax is way nicer than AS syntax.
Then once its good and tight i can write it in asm!

You bet.
I'm making an assembler with multiple targets about now, and here are some stats:
-Supporting code: 1100 LOC
-Code generator for ZPU microcontroller: 200 LOC (and complete)
-Code generator for a memory-based VM: 250 LOC (and also complete)
-Code generator for x86-32: 1250 LOC (and still barely scratches the surface)

To make one you need to carefully study the instruction set, which have a few patterns, and design from ground up with ALL the cases in mind. Then you'll get a fully functional assembler when everything will be done. Many asm-for-fun projects simply run out of steam on first encounter with mov al, [ds:si+bx] or something.

MSA a simple DOS assembler with a NASM-like syntax in about 1500 lines of C.
At first glance seems to runs fine under xp, i used old Turbo C++ 3.0 to compile sources.

MSA

A86

It depends: 1250 LOC C or 1250 LOC ASM?

Very cool, i will study these sources now.
And i have Turbo C 2.0 also on my dev box.
Aaah, the good old days, where makefiles ruled.

8086 Assembler Implemented in C++. Single source file.

There are a lot of assemblers written in Forth for various processors, and they tend to be very small because they use Forth's interpreter rather than do their own parsing. As Chuck Moore famously said: "Let the dictionary do the deciding."

You could start with LMI UR/Forth, which is available in both 16-bit and 32-bit. It used to be a commercial product. Now it is shareware, although Laboratory Microsystems Inc. is out of business afaik. A lot of it is open-source, including the assembler.

The company where I used to work had the complete source for UR/Forth, but I never saw it because I hadn't signed the non-disclosure form. There are other Forths available for the 16-bit x86 that are fully open-source that you might be able to port over to your OS.

Why do you want an assembler that runs under your OS? What advantage would that have over using FASM to cross-assemble? On the other hand, getting a Forth to run under your OS would make sense, as you would get the Forth interactive development environment.

Why are you targeting the 16-bit x86? Did you buy a computer at the Salvation Army for $5 and decided to write an OS for it?

Here's another small 8086 Assembler which i discovered by chance, written in C by Dave Dunfield, about 1400 lines.

MICRO 8086 ASSEMBLER

There is too the as86(can run even on Linux) and M80(by Microsoft) . But both (until I know) it is'nt free. Some older OS such as CP/M provide an assembler for 8086. Also,I'm not sure,but I think that the BCC(Bruce's C Compiler) can act as an assembler too.

bitshifter, IMHO, if you want to write OS, you must concentrate on the OS, not on the assembler. You can simply port FASM for your OS and to have very powerful assembler with only 2..3 days coding. Then you will be free to create your fine OS and you will have decent assembler from the day one.

http://posx86.codeplex.com/SourceControl/latest#programs/pasm.asm
Checkout PASM. Its for my OS.
16 bit 8086 Assembler for Posx86