Somebody knows, on how parameters in function which work through FPU under Win64 are transferred and come back?

There is no standard way to use x87 FPU under win64. For FP numbers, you should use low part of XMM registers or pointers to such parameters in memory.

EDIT: here is a reference with the official explanation:

http://msdn2.microsoft.com/en-us/library/ms235286.aspx#sectionSection0

not well documented, but get by playing:
don't forget to use emms or femms before you put something into fp reg
you may use xmm and fp regs together, but not mm and fp

this is not standard way, but works well for me in case procedure doesn't contain any API and doesn't access any mm reg (API may use mm reg and this destroys fp immediately):

femms ; (or slow emms for compactibility with intel CPU)
fld something
call procedure
...

When your proc uses fp and API, you have to use the way described by MazeGen
- put fp in the stack space, by memory pointer, in xmm reg (APIs use and destroy xmm0-5, so use xmm6-15 which are protected). The same way is for return from proc.

Thanks to all, it has very much helped me.