In addition to outputting EXE and DLL files, is it also possible to output win32 object files (aka COFF files) to be linked via an external linker? In general, where can I find a list of available output formats?

All existing formats are available: pdf, bmp, mp3, wav, etc. not only executable formats.
If thou asking about predefined formats in fasm1? so they are PE and PE+ (file extension on thour taste EXE, DLL, SCR, ...), pure MZ - (extension too EXE but only DOS header in, no new exe header such as PE inside). then COFF & MSCOFF and their 64 bit versions. then ELF for linux in variations as object or as executable in both 32 & 64 bit variant.
In case of fasmg there is no predefined formats but we could speak about formats that thou will have out of the box (they already realized in fasmg in official package or in topics of this forum).
In addition to fasm1 predefined formats ara machO format for executables and objects but only in 32 bit form. Described in begin of topic pdf, bmp, wav & mp3 are already realized in fasmg too.
java js files realized too.
I missed COM - it is exists in fasm1 & fasmg too.

In case of fasm1 to formats exist one limitation: in one file couldn`t be intermixed sections & segments - so machO couldn`t be realized and rare cases of elf where sections and segments can be mixed can`t be realized too.
But if thou not going to make general macros for building machO - if thou ready to manualy calculate segment & sections offsets - thou could create even machO in fasm1 - but one editing of code could force thou recalculate this things manualy again.

In case of windows thou could create even activeX files with COM objects inside.

So in the FORMAT line at the start of an ASM file intended to be compiled with FASM, what are the available names I can specify for the format. Is there a list of these somewhere?

subchapter 2.4 formatter directives of file "fasm.pdf" (if thou download fasm - it will be inside too)

format binary - any format that thou build manualy. others are predefined.

Or you can just browse the section 2.4 on this website.