flat assembler
Message board for the users of flat assembler.

Index > Projects and Ideas > Ultra micro data base engine?

Author
Thread Post new topic Reply to topic
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
So, I need UMDBE ( Very Happy) in order to save and restore different program settings, options, some data, etc.
Something like .INI files, but proper parsing of .INI files is too hard to be implemented in small package.

There are several requirements:
1. Flexible enough to keep different types of data - numbers, strings and byte arrays.
2. The engine that provides read/write access to the database should be very small - 1kbyte is too big, 512bytes - OK.
3. The engine should be portable enough among different OSes. The OS should provide only general file access and eventually memory allocations.

There is no requirement for human readable file format.

I have some ideas about the file format, but please, share your thoughts about the subject.
Post 03 Jan 2012, 09:02
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
typedef



Joined: 25 Jul 2010
Posts: 2913
Location: 0x77760000
typedef
Are we talking about a binary file w/ executable code or an actual program, or a data file format.

IMO,file format is the easiest.

For example let's make pseudo format here.

Code:
// the header

records dd 0  ; Number of records
record_1
  size dd  ? ; size of it
  data dd ? ; the size of the data
end_record
...
more records...

    


now when reading and updating, you just adjust the file pointer to the exact record and re-write to it.
Post 03 Jan 2012, 10:29
View user's profile Send private message Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1419
Location: Toronto, Canada
AsmGuru62
I'll see if I can help -- I have some ideas for my own Settings Manager, I can share. All my data is at work, so I'll send some ideas from work.
Post 03 Jan 2012, 11:22
View user's profile Send private message Send e-mail Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
typedef, it is not as simple as it sounds. Of course you always can save to the file some memory block and then to read it back. The trick is that the data have to be accessed by its identifier, so if you change the data format of the memory structure, the saved data still to be readable and useful.
Post 03 Jan 2012, 11:48
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1419
Location: Toronto, Canada
AsmGuru62
I am planning to use the following design.
The data file will have two parts:

1. Contents
2. Records of data

Both parts will be sector-aligned in size, since I
decided to use the FILE_FLAG_NO_BUFFERING to get/set the data.

PART #1 (Contents) will represent a vector of 8-byte structures
like the one below:
Code:
struct CONFIG_ITEM
        DataType        UINT16  ?       ; Integer,String,Binary
     ByteSize        UINT16  ?       ; To use in ReadFile() / WriteFile()
        ByteOffset      UINT32  ?       ; To use in SetFilePointer()
ends
    

As an example, say, Contents has 4 sectors in size -- this will
provide #items = (512*4)/8 = 256. So, it will have room for 256 setting entries.
Basically, if I need the settings with ID=35 - it will be a direct index into that vector.
From the structure I will know where in the PART #2 my data is located and its size.
To make code more readable - I will define some values, like:
Code:
CFG_PARAMETER = 35
    

PART #2 (Records) will immediately follow 1st PART and contain 32 byte records of whatever data
is defined in Contents. I know, some disk bytes will be wasted, but disks are cheap these days
and I need to work in sectors, so all records are the same size of 32 bytes for now.
Code:
struct CONFIG_RECORD
 Data    BYTES   32
ends
    

Code is not written yet.
It is just design.

At program startup the 'DataType' and 'ByteSize' members for each CONFIG_ITEM will
be set up from a static .data vector, like the one below:
Code:
ConfigMap dw  CFG_BINARY, sizeof.RECT,\
          CFG_STRING, 256,\
          CFG_INTEGER, 4,\
           etc.
    

and then 'ByteOffset' will be calculated at run-time using 'sizeof.CONFIG_RECORD'.
If DB file is missing -- I'll create an empty file with
all data set to zero and then set some default values as needed.
Post 03 Jan 2012, 19:01
View user's profile Send private message Send e-mail Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
AsmGuru62, the parameters will be identified by it's position in the contents section, right?
In my opinion, this can lead to problems, when the new parameters are inserted and some of the old parameters are not used anymore. In this case, all parameter numbers have to be changed.
...
For now, I tend to use some version of RIFF-like format. I am using this format for the Fresh project files and it seems to be flexible enough to keep whatever data you need. The identification of the data records is based on some 4-byte human readable ascii "names", so the place of the record in the file is not important.
The format of the Fresh project file is described here
What you guys think about it?

Regards.
Post 04 Jan 2012, 07:19
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1419
Location: Toronto, Canada
AsmGuru62
Well, the settings file is pretty much an unchangeable thing.
I will be only adding values to it and not removing them.
If some setting item will be no longer needed (very rare case) -- I will
simply leave it in a file.

I was also thinking of making a huge structure with all the settings and simply writing/reading it in one shot. Also, will work fine.
Post 04 Jan 2012, 11:28
View user's profile Send private message Send e-mail Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
AsmGuru62 wrote:
I was also thinking of making a huge structure with all the settings and simply writing/reading it in one shot. Also, will work fine.


Exactly this was my approach in the case with Fresh themes (editor coloring and other settings) and with Fresh project files. Of course it works just fine, but this approach is big mistake!
It makes things really hard coded and every change in the format needs some converter program to convert from old format to the new format. Another possible solution is to support both old and new formats in your program, i.e. to write over-bloated code.
Post 04 Jan 2012, 12:37
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1419
Location: Toronto, Canada
AsmGuru62
I solve it with adding all new stuff ALWAYS at the end of the structure.
When I call ReadFile() -- it will read a new size, but if the old file was open -- only the first bytes are loaded, but ReadFile() will not fail.
It works perfectly.

1. Set all members to default values
2. Open file (old or new, does not matter)
3. Load #bytes = sizeof.<NEW_STRUCTURE>

If there was not enough bytes in a file -- I end up with the new
data members at the end of the structure set up with default values,
so code using these members does not fail. When the structure is saved -- this file now will have all the bytes, because structure increased in size.
Post 04 Jan 2012, 16:16
View user's profile Send private message Send e-mail Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
OK, here is the first more or less functioning version of the uConfig library. It became bigger than I though - 1200bytes, but it have more features. Smile
Actually it is implementation of simple hierarchical data base, similar to the Windows registry.
You can check or download the sources of the library online. At the begin of the source file, there is a description of the database file format.
Also, I attached example of simple database editor for uConfig databases:
"uConfigEditor.exe" for Windows
"uConfigEditor" for Linux. (You probably should set the permissions to "exec" for Linux version.)
Also simple example uConfig database is there for tests.

As long as this is the first more or less useful application created with FreshLib, please test it and report the "show stopper" bugs.
..
There are many known issues with the controls - for example you can't select text in the edit boxes with the mouse. There is no clipboard as well.


Description: uConfigEditor portable
Download
Filename: uConfig.zip
Filesize: 37.04 KB
Downloaded: 433 Time(s)

Post 24 Jan 2012, 10:00
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.