ELF, ELF64 and use32?

I'm trying to get a 64bit kernel to be multiboot-compliant, with a bootstrap code to get it into long mode.

For that there is a crt0.o in fasm and the rest in HLL.
For 32 bit case there were no problems, but 64 bit case don't quite want to play nice.

It starts something like that:

Code:

format ELF64
section '.text' executable
;//###########################################################################//
;magic stuff to enable multi-boot compliance...
mb_magic:            dd 1BADB002h
mb_flags:           dd 3
mb_checksum:        dd 0-3-1BADB002h
;//###########################################################################//
use32
_start:
 cli
 lgdt   [gdt32_limit]
 lidt  [idt32_limit]
 jmp   next

But the use32 is apparently ignored, since lgdt gives a size error.

Meanwhile, if i use format ELF then the LD linker won't see it as valid, but it compiles ok, and use64 appears to be used.

So, is there any way to do this nicely?
32-bit & 64-bit-code o + HLL code o into one multiboot file.

Also, would GRUB read ELF64?