object files and libraries

For the COFF format the best reference is the official PE/COFF specification from Microsoft.
There is also this nice old website about the DJGPP COFF variant if you'd like something simpler to begin with. Note that DJGPP COFF (generated by fasm with "format COFF" setting) is generally a subset of modern Microsoft's COFF (generated by fasm with "format MS COFF" setting), with some minor differences in how relative relocations are computed.

As for the ELF format, the original specification, though dated, should still suffice (with supplements for things like x86-64 ABI).