Suggestion: Save Virtual Block as File

Index > Compiler Internals > Suggestion: Save Virtual Block as File

Goto page Previous 1, 2, 3 Next

Author

Thread

shutdownall

Joined: 02 Apr 2010
Posts: 517
Location: Munich

shutdownall 19 Jun 2014, 16:33

I think this would be a great feature to save some data to one or more several files by moving the desired data in a virtual block.
Address space label would be fine for this.

This would offer maybe nice custom debug and protocol functions as well.

Directories can be used in the same way as in the load file directive. No need for complicate checks. If write file fails - it's the users fault.

For security issues there can be allowed relative subdirectories from FASM path or source path only - no absolute path values like /etc/passwd.

And for the performance during multiple passes - I had the same issue when adding some features to FASM Z80 version with allowing forward references. I just waited till the last pass and then force one addition "final pass". Maybe that would be a good idea to wait till code is resolvable and than add one additional pass for things you only want to do one time per compilation. Lets call it final pass to distinct it from last pass.

So I find it great and please Tomasz, go on for realisation.
If this feature comes it would force me to update the Z80 version to the latest FASM version because it could become very useful. Cool

19 Jun 2014, 16:33

evk1

Joined: 18 Jun 2014
Posts: 24

evk1 19 Jun 2014, 17:36

Quote:

For security issues there can be allowed relative subdirectories from FASM path or source path only - no absolute path values like /etc/passwd.

Code:

../../../../../../../../etc/passwd

_________________
Sorry for my English

19 Jun 2014, 17:36

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 17:37

I think that in case of fasmw/fasmd this could be handled in a better way - in the window containing the compilation summary (including the DISPLAY data) they could present the list of generated "virtual files" and allow to save some or all of them into file system under the same names. For console version there would be only option to automatically convert all virtual files into a real ones upon finishing, or otherwise discard them.

shutdownall wrote:

For security issues there can be allowed relative subdirectories from FASM path or source path only - no absolute path values like /etc/passwd.

Perhaps restricting to directive to specifying the file name only, without any path elements, would be enough. The "virtual files" in principle need only name. And when choosing to save individual "virtual files" into file system one could choose (in a "Save as"-style dialog) a different path if needed (the default would be the same as for output).

shutdownall wrote:

And for the performance during multiple passes - I had the same issue when adding some features to FASM Z80 version with allowing forward references. I just waited till the last pass and then force one addition "final pass". Maybe that would be a good idea to wait till code is resolvable and than add one additional pass for things you only want to do one time per compilation. Lets call it final pass to distinct it from last pass.

I used such additional "final" pass in the earliest versions of the listing feature, but did not like this solution and it introduced some problems that I wanted to avoid.
Anyway, the "virtual files" design I mentioned above would intrinsically avoid the problem of redundant file system accesses.

shutdownall wrote:

So I find it great and please Tomasz, go on for realisation.
If this feature comes it would force me to update the Z80 version to the latest FASM version because it could become very useful.

I knew you would be among the ones to appreciate such addition.

19 Jun 2014, 17:37

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 18:07

m3ntal

Quote:

What's wrong with specifying a folder like \BINARY\ or ..\? "file" doesn't have this limitation. For security?

Yes.

Quote:

How do you ensure the same security at runtime?

I can't. For that reason I disallow myself to start non-trustworthy code. But at least a can try to compile it without any fears. So why adding an additional attack vector, and make even the compilation dangerous?

The whole information security world tries to reduce the number of possible attack vectors, and you are suggesting the opposite.

Quote:

Quote:
those are rather tasks for some scripting languages...
Umm... if I'm not mistaken, FASM language is an interpreter.

You took the citation out of context. "Some scripting languages like shell or autohotkey", i.e. those intended to automate OS-wide and OS-level tasks. The fasm macro-language is meant to script the compilation process only.

Tomasz Grysztar

Quote:

The behavior of this directive would not affect the output of assembly, hence the SSSO principle is irrelevant here

It would. At least on the level of multiple successive compilations. A prescription for compiling a program could be: "Compile it two times in order to ..." . The output of the second compilation may depend on whether the command line allows the new store directive. Is there any advantage at all in allowing arbitrary destinations? I mean, why would you want to add this additional burden of risks estimation required to venture upon enabling the new store directive with a command line switch?

Quote:

unless the assembler wrote the contents of file early enough that it could affect some LOAD - but I think such behavior should be undefined and this kind of usage discouraged

I don't see any reasons for that. Why?

Quote:

Probably the best option would to buffer all the data and write it at the same time when the DISPLAY buffer is flushed

That is exactly what I suggested before. This way you avoid multiple overwriting of the same file not only across consecutive passes but also within a single pass. What is more important here is to discard the output on compilation failure as opposed to how the display directive works.

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 18:07

m3ntal

Joined: 08 Dec 2013
Posts: 296

m3ntal 19 Jun 2014, 18:09

shutdownall:

Quote:

No need for complicate checks. If write file fails - it's the users fault.

Isn't it dangerous to just overwrite certain files without checking for existence? Not saying that FASM should have this feature.

Quote:

Directories can be used in the same way as in the load file directive... no absolute path values

"file" directive supports absolute paths.

(PS: This entire discussion reminds me of gun control arguments. You can't have a gun for protection because some insane kid might go kill random people. It's more like saying police officers can't even have guns. We are experienced programmers who know what to do. Just because some beginners might misuse it is no reason why we can't use it).

_________________
New FASM Site, Examples, Graphics, Updated Libraries

19 Jun 2014, 18:09

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 18:19

l_inc wrote:

It would. At least on the level of multiple successive compilations. A prescription for compiling a program could be: "Compile it two times in order to ..." . The output of the second compilation may depend on whether the command line allows the new store directive. Is there any advantage at all in allowing arbitrary destinations? I mean, why would you want to add this additional burden of risks estimation required to venture upon enabling the new store directive with a command line switch?

The SSSO rule cannot be applied to the cases when the source files get modified (and the binary includes are also the source, fasm's FILE is what in other assembler is usually called INCBIN). When you use STORE to modify one of your sources, the "Same Source" part is no longer true. The same reasoning could be applied to the case of fasm-script altering its own source file (like "format binary as 'asm'").

l_inc wrote:

Quote:
unless the assembler wrote the contents of file early enough that it could affect some LOAD - but I think such behavior should be undefined and this kind of usage discouraged

I don't see any reasons for that. Why?

Because this would limit possible implementations and it does not give any advantages that would be worth it (if you need to store and then load the value, just use the virtual block, file system does not need to be involved).

19 Jun 2014, 18:19

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 18:22

m3ntal wrote:

shutdownall:
Quote:
No need for complicate checks. If write file fails - it's the users fault.
Isn't it dangerous to just overwrite certain files without checking for existence? Not saying that FASM should have this feature.

fasm always overwrites the output file without asking. You can run "fasm my.asm my.asm" command and fasm will happily overwrite your source file with the output of assembly.

19 Jun 2014, 18:22

m3ntal

Joined: 08 Dec 2013
Posts: 296

m3ntal 19 Jun 2014, 18:46

Mr. Inc:

Quote:

The whole information security world tries to reduce the number of possible attack vectors, and you are suggesting the opposite.

I'm sure you'll feel safer, but am not convinced that a limited STORE will have any effect on the "whole information security world". How many *active* FASM users are there in the world? Probably not even 500. How many C++/Java/script programmers? Multi-millions. If you were really concerned about safety/security, you'd be talking about Javascript.

Source can contain hidden malicious code. Example: Is this a font or a virus?

Code:

font: db 11001100b, 1111000b, 10101010b, ...

My point: If one wants to create a virus/spyware, they'll do it regardless and there's nothing you, I or anyone else can about it.

19 Jun 2014, 18:46

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 19:03

Tomasz Grysztar

Quote:

When you use STORE to modify one of your sources, the "Same Source" part is no longer true.

No, it is. You are missing that double compilation is viewed as a single combined operation. The original source is the same. But the output of the double compilation is different depending on whether the new store is allowed.

Quote:

The same reasoning could be applied to the case of fasm-script altering its own source file (like "format binary as 'asm'")

It cannot. Because there is no command line switch to disallow compiling into its own source. Btw. I used that behaviour once in order to achieve register randomization. It was a miserable attempt though.

Quote:

Because this would limit possible implementations

Why do you need multiple possible implementation possibilities if there's only one which is good?

Quote:

and it does not give any advantages that would be worth it

It does actually. As soon as you do a store of an addressing space to a file, you can alter the addressing space, but still have a possibility to access its older saved state by accessing the file. Even if it only exists in memory (see my "redirection" suggestion above).

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 19:03

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 19:06

m3ntal

Quote:

Mr. Inc

How do you think, what's the first letter of my nickname?

Quote:

How many *active* FASM users are there in the world?

Do I understand it correctly that you purpose is to endanger the fasm users only?

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 19:06

m3ntal

Joined: 08 Dec 2013
Posts: 296

m3ntal 19 Jun 2014, 19:21

l_inc: I can't remember whether it's 1, l or I. They use the same characters on my phone browser. No internet at home. Must go outside with phone.

Quote:

Do I understand it correctly that you purpose is to endanger the fasm users only?

Yes, by making dangerous suggestions like this Wink

19 Jun 2014, 19:21

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 19:26

l_inc wrote:

Tomasz Grysztar

Quote:
When you use STORE to modify one of your sources, the "Same Source" part is no longer true.

No, it is. You are missing that double compilation is viewed as a single combined operation.

Is viewed by whom? I see it as a batch-like operation. And the result is then highly dependent on the contents of the batch commands you used. If in the first call to fasm you specified the output to be the same file that is the source for the second command, the result will be different compared to the case when the first command is modified to use different file as output.

l_inc wrote:

Quote:
The same reasoning could be applied to the case of fasm-script altering its own source file (like "format binary as 'asm'")

It cannot. Because there is no command line switch to disallow compiling into its own source.

But you have a command line option (that is: the output field itself) that allows you to choose compiling into own source or not. See above.

l_inc wrote:

Quote:
Because this would limit possible implementations

Why do you need multiple possible implementation possibilities if there's only one which is good?

Well, the one that is good is actually the one that has no file writes during assembly at all. So if this behavior is to be defined exactly, it will be the other way round. Initially I was not sure of what implementation I would choose, and I considered that I may even switch implementations later, that's why I spoke of "undefined" instead.

l_inc wrote:

It does actually. As soon as you do a store of an addressing space to a file, you can alter the addressing space, but still have a possibility to access its older saved state by accessing the file. Even if it only exists in memory (see my "redirection" suggestion above).

You can copy the content of addressing space to another virtual block and even label it with a file-like name. Doing the same thing but through disk instead of memory seems ugly.

19 Jun 2014, 19:26

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 20:56

Tomasz Grysztar

Quote:

Is viewed by whom?

By me in my argumentation. And I assume you use its context when answering. But actually you are right about the batch-like operation and the possibility to override the default output file name using the command line. Thus I think you can disregard this one.

Quote:

Well, the one that is good is actually the one that has no file writes during assembly at all. So if this behavior is to be defined exactly, it will be the other way round.

It's not the other way around. It's the same way I was talking about from the beginning: store files in-memory until the compilation is successfully completed and redirect any file and include accesses to the memory during the compilation. I mean it's just an interface question, not an implementation one. Or is there any technical problem about creating a hash table of file names, that is checked before trying to access the file system? Btw. it seems, you agreed on the "file names only" limitation for the new directive, did you?

Quote:

Doing the same thing but through disk instead of memory seems ugly

I think the ugliness here is determined solely by an inefficient implementation. From the interface point of view I store a file, because I want to store the file, and I access the file because I want the state of the file, when it was last stored, not the current state of the addressing space. A necessity to create an additional addressing space for that does not seem to be justified in any way.

m3ntal

Quote:

Yes, by making dangerous suggestions like this

You got the point. Smile

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 20:56

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 21:43

l_inc wrote:

It's not the other way around. It's the same way I was talking about from the beginning: store files in-memory until the compilation is successfully completed and redirect any file and include accesses to the memory during the compilation.

That would be a specific implementation and I did not want to decide that at the moment. For the reason that I may prefer implementation that would postpone changes in files until the moment of writing the complete output (so that LOAD and INCLUDE would never be affected) I did not want to define such behavior at all. But now, as I decided to go with postponed write, I can specify this: the defined behavior would be the opposite one, the files in file system (as used by LOAD/INCLUDE) would never be modified during the assembly. The "virtual files" created by STORE AS would proceed to be written into file system after the compilation is finished, and potentially under a different names than the ones specified in source - in IDE versions user may be allowed to choose where and under what name to save each one of them.

l_inc wrote:

Or is there any technical problem about creating a hash table of file names, that is checked before trying to access the file system?

I see one potential problem: fasm might not be able to determine whether the path given to LOAD directive leads to the same file that it has buffered in memory. Depending on a specific OS and file system features the same file can be specified in a multitude of ways. fasm's simple OS abstraction layer has no way to deal with this kind of problem.

l_inc wrote:

Btw. it seems, you agreed on the "file names only" limitation for the new directive, did you?

When I invented the concept of defining such directive as creating the "virtual files" that can later be stored somewhere or not, depending on user's choice, that limitation seemed natural. Since there would be a plain list of "virtual files" created, like a supplement to the basic DISPLAY buffer, each one of them would just need a name - a path would be a bit out of place there, IMO.

l_inc wrote:

Quote:
Doing the same thing but through disk instead of memory seems ugly

I think the ugliness here is determined solely by an inefficient implementation. From the interface point of view I store a file, because I want to store the file, and I access the file because I want the state of the file, when it was last stored, not the current state of the addressing space. A necessity to create an additional addressing space for that does not seem to be justified in any way.

Creating a file for a temporary storage when the same thing could be done safely within memory appears ugly to me, no matter whether it is assembly or some other language. Even if it does not impact the performance, I do not like files polluting the place. But this may be a matter of taste. Smile

19 Jun 2014, 21:43

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 22:23

Tomasz Grysztar

Quote:

the files in file system (as used by LOAD/INCLUDE) would never be modified during the assembly

Any advantages of this definition with respect the opposite one? An obvious disadvantage is in the last paragraph of this post.

Quote:

The "virtual files" created by STORE AS would proceed to be written into file system after the compilation is finished, and potentially under a different names than the ones specified in source

Don't you think, that load or include should be able to access the virtual files at least between compilations? If you do, how do you plan to provide this possibility to these directives?

Quote:

fasm might not be able to determine whether the path given to LOAD directive leads to the same file that it has buffered in memory

Hence the question above.

Quote:

Creating a file for a temporary storage when the same thing could be done safely within memory appears ugly to me

I think you are incorrectly sticking to the files as persistent storage only, even though you defined the "virtual files" as something that is even not necessarily stored (when using the IDE). The file abstraction is very often used for many things such as memory mappings, pipes, devices etc. If you consider this and extend your "virtual files" abstraction just a tiny step forward, it might not appear ugly to you anymore.

But I actually meant something different. Once again, "I store a file, because I want to store the file", not because I want some temporary storage. And after that (by a magic coincidence) I want to access the state of the addressing space when the file was stored. So why would you want to force me to create an additional copy of that addressing space, if I already have it in the form of the "virtual file"?

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 22:23

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 19 Jun 2014, 22:51

l_inc wrote:

Don't you think, that load or include should be able to access the virtual files at least between compilations? If you do, how do you plan to provide this possibility to these directives?

What do you mean by "between compilations"? All my considerations apply to a single compilation only - after it you have just a bunch of files (or not - depending on your settings). What you do with them later is up to you, just like in a batch file where you can do a lot of file shuffling between the consecutive calls to compiler, and it does not concern fasm.

l_inc wrote:

I think you are incorrectly sticking to the files as persistent storage only

Why incorrectly? Treating files this way is one of the assumptions of fasm's OS abstraction layer. I do not want to deal with things that apply only to select OS designs. Thus for the sake of this discussion, when I talk about "file" I mean persistent storage.

l_inc wrote:

But I actually meant something different. Once again, "I store a file, because I want to store the file", not because I want some temporary storage. And after that (by a magic coincidence) I want to access the state of the addressing space when the file was stored. So why would you want to force me to create an additional copy of that addressing space, if I already have it in the form of the "virtual file"

The way I defined it now, you don't "have it". With FILE you access the source, while the "virtual file" resides in output. It is like wanting to access the contents of DISPLAY buffer because it contains some data you need and maintaining another copy seems wasteful - we could talk about adding some new ways to access such buffers, but there would have to be some actual usage cases where it really pays out.

PS In my post above I wrote about LOAD/INCLUDE, when I did actually mean FILE. LOAD is no longer able to access files (this feature was removed in fasm 1.52), and INCLUDE is handled by preprocessor, long before the assembler module is started.

19 Jun 2014, 22:51

l_inc

Joined: 23 Oct 2009
Posts: 881

l_inc 19 Jun 2014, 23:16

Tomasz Grysztar

Quote:

What do you mean by "between compilations"?

I mean running the console fasm version with identical command line arguments twice without involving any additional tools. I mentioned two use cases for the new store directive before: build number incrementation and code randomization. For these use cases it's important that the file or the include directive accesses the same file stored by the new store directive during the previous compilation. If you invalidate these and similar use cases, then what's the whole topic about?

Quote:

Treating files this way is one of the assumptions of fasm's OS abstraction layer

Not after you defined "virtual files" as something that may or may not become the persistent storage file, right?

Quote:

I do not want to deal with things that apply only to select OS designs

My comments are about the design of new abstractions you define and involve while designing the new feature of fasm. The mentioned OS abstractions are just for the purpose of comparison.

Quote:

It is like wanting to access the contents of DISPLAY buffer because it contains some data you need and maintaining another copy seems wasteful

The probability of needing access to the display buffer is too low. There are no real use cases I can think of. However the "virtual file" output is very likely to become a source file for subsequent compilations. That is why the suggestion to be able to access the file within the same compilation seems to me very natural.

_________________
Faith is a superposition of knowledge and fallacy

19 Jun 2014, 23:16

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20299
Location: In your JS exploiting you and your system

revolution 19 Jun 2014, 23:40

I wonder how this would affect fasm as DLL?

19 Jun 2014, 23:40

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 20 Jun 2014, 00:06

l_inc wrote:

Tomasz Grysztar

Quote:
What do you mean by "between compilations"?

I mean running the console fasm version with identical command line arguments twice without involving any additional tools. I mentioned two use cases for the new store directive before: build number incrementation and code randomization. For these use cases it's important that the file or the include directive accesses the same file stored by the new store directive during the previous compilation. If you invalidate these and similar use cases, then what's the whole topic about?

How would I invalidate these? Every time fasm is called it is a completely separate process, taking as a source the files existing in OS after the previous one has finished. Do you think that fasm would somehow be able to prevent its other instance in the future from accessing some of the files it created? I'm confused.

l_inc wrote:

Not after you defined "virtual files" as something that may or may not become the persistent storage file, right?

The "virtual files" are not actual files in my understanding, just like guinea pig is not really a pig.

20 Jun 2014, 00:06

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 20 Jun 2014, 00:10

revolution wrote:

I wonder how this would affect fasm as DLL?

Some additional buffers would need to be passed through the API. Current API variants would then just ignore the additional outputs completely.

20 Jun 2014, 00:10

Goto page Previous 1, 2, 3 Next

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum