flat assembler
Message board for the users of flat assembler.

Index > Heap > Petabox and the death of NTFS

Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 10, 11, 12  Next
Author
Thread Post new topic Reply to topic
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
Wasn't it about 16 exabytes or something?
Erm, when was 16EB mentioned?
Borsuc wrote:
But the main concerns here are probably not usage (although I think it will be) but physical limits.
Have confidence. Such things like the existing interfaces are not set in stone, they can be changed, upgraded, improved, enhanced also. We have already moved from ATA to PATA/SATA in just a few years, no reason not to have a next generation interface. Maybe it will be FATA, as in 'Fast' ATA?
Post 01 Apr 2009, 21:48
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
Erm, when was 16EB mentioned?
The NTFS limit? I don't know anything about it though.

revolution wrote:
Have confidence. Such things like the existing interfaces are not set in stone, they can be changed, upgraded, improved, enhanced also. We have already moved from ATA to PATA/SATA in just a few years, no reason not to have a next generation interface. Maybe it will be FATA, as in 'Fast' ATA?
Not digital, or not serial. But then again even a 3D structure with 3nm data bits has to be 10x10x10cm in volume (waaay too big) to hold around a few exabytes. As for petabytes -- I don't think hard drives are really that efficient at the space storage. I'm not saying we won't have something else instead -- personally I see solid-state disks as taking the trend in the future, but we all know they are very expensive these days (compared to HDDs obviously).

This would be like people in 1990 saying that we would have 10Ghz by now. See for yourself why it has taken a different path, because of "physical difficulties" to name it in a simple way.

_________________
Previously known as The_Grey_Beast
Post 01 Apr 2009, 23:02
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
But then again even a 3D structure with 3nm data bits has to be 10x10x10cm in volume (waaay too big) to hold around a few exabytes. As for petabytes -- I don't think hard drives are really that efficient at the space storage. I'm not saying we won't have something else instead -- personally I see solid-state disks as taking the trend in the future, but we all know they are very expensive these days (compared to HDDs obviously).
I think your calculation is not correct. I'm not sure what you mean by "a few exabytes" But I will assume you mean 16EB which you mentioned above. 2^68 bits (2^67 plus say doubling for other logic and control) in a cube with equal sides is 6658043 units of 3nm per side. Gives 20mm (or 2cm) per side.
Borsuc wrote:
This would be like people in 1990 saying that we would have 10Ghz by now. See for yourself why it has taken a different path, because of "physical difficulties" to name it in a simple way.
So, people make all sorts of predictions, some turn out correct and some incorrect. But that has no bearing on future predictions. I am not predicting clock speeds. I am predicting storage availability.

Perhaps we can work it backwards. Let's take the existing 3.5" HDD (101.6 mm × 25.4 mm × 146 mm) dimensions and see how big each bit can be and still get 1PB inside (again assuming a doubling for other logic, control, connectors etc.).

Cubic capacity: 376773.44 mm^3
Total bits: 2^54 (1PB+overhead)
Cubic space per bit: 2.09e-11 mm^3
Linear space in a 3D bit: 2.755e-4 mm (~275 nm)

So with a bit size of 275nm per side we can build a 1PB storage device quite comfortably. Of course, we are not there yet with 3D technology. And the 3D grid would be 369455 x 92364 x 530910, which is quite a daunting thing to imagine. But the bit size is actually large compared to the current capability of microelectronics. So if/when the breakthrough comes with a nice small (say 200nm) 3D stackable bit storage technology then 1PB should be no problem.
Post 02 Apr 2009, 06:55
View user's profile Send private message Visit poster's website Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4633
Location: Argentina
LocoDelAssembly
All the calculation are based on one bit per "cell"? What about a MLC-like storage?
Post 02 Apr 2009, 16:14
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
LocoDelAssembly wrote:
All the calculation are based on one bit per "cell"? What about a MLC-like storage?
Well, sure, that can only help to increase the storage capacity or allow larger "cell" sizes and still meet the 1PB requirement.

But I was just trying to put it into perspective about the relative sizes of things. If something like the 3D memory structures can be made practical then we could be looking at some very large (by today's standards) capacities in the future.

There is an episode of Star Trek: Enterprise where the bad guys delete a body of information from the ship computer and it shows on the screen "Volume free space: 19.3 XB". So you see it will happen before the year 2250 or so. :p (although I am not sure what XB actually means but I am assuming in the future that XB becomes the popular notation for EB)
Post 02 Apr 2009, 16:28
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
I think your calculation is not correct. I'm not sure what you mean by "a few exabytes" But I will assume you mean 16EB which you mentioned above. 2^68 bits (2^67 plus say doubling for other logic and control) in a cube with equal sides is 6658043 units of 3nm per side. Gives 20mm (or 2cm) per side.
Hmm could it be that I actually skipped one unit in the meantime?
And by "a few" I meant 32 actually.

http://en.wikipedia.org/wiki/Zettabyte

Razz
revolution wrote:
I am not predicting clock speeds...
Hmm how do you think Serial interfaces like SATA work? Wink

Yeah sure, maybe we'll have harddrives integrated into the RAM or get rid of HDDs altogether and just use non volatile RAM. But like I said, it would be different & change of design anyway. NTFS or not, using RAM both for memory and HD will change dramatically all software design anyway.

_________________
Previously known as The_Grey_Beast
Post 02 Apr 2009, 23:02
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
Hmm could it be that I actually skipped one unit in the meantime?
And by "a few" I meant 32 actually.
Okay, so for each doubling the figures increase by ~1.26. 2cm ---> 2.52cm (eerily close to 1 inch!). Using your figures, 10cm^3 is ~2ZB . But why are you choosing 32EB?
Borsuc wrote:
Hmm how do you think Serial interfaces like SATA work? Wink
I am only predicting storage space, not the access method. Although I do consider things like P/SATA will not be adequate for future large drives.
Borsuc wrote:
Yeah sure, maybe we'll have harddrives integrated into the RAM or get rid of HDDs altogether and just use non volatile RAM. But like I said, it would be different & change of design anyway. NTFS or not, using RAM both for memory and HD will change dramatically all software design anyway.
But that was my point actually. Glad to see you agree with me. Once the large storage comes in, NTFS will die.
Post 03 Apr 2009, 01:27
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
revolution wrote:
But that was my point actually. Glad to see you agree with me. Once the large storage comes in, NTFS will die.
Or simply evolve - like EXT2->EXT3->EXT4. And nothing stops Microsoft from doing something completely new (akin to EXTx -> BTRFS) and still calling it NTFS Smile (or perhaps NTNTFS Razz).

_________________
Image - carpe noctem
Post 03 Apr 2009, 03:17
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
NTFSjr

But think it will require a little more than a simple tweak to make NTFS usable on 1PB storage media, or SM (I decided that "drive" is not an appropriate term as there is a good chance all mechanical parts will be stationary).

Besides, MS can't properly market it if it just looks like just a minor change, they will want to hype it to the maximum (regardless of how much change there really is) and call is something fancy and new that is meant to be catchy. So the standard marketing nonsense will apply and we will get a new FS and a new name.
Post 03 Apr 2009, 04:13
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
But why are you choosing 32EB?
Sorry, in my previous post, I meant 32ZB that's why I put the link Razz

revolution wrote:
I am only predicting storage space, not the access method. Although I do consider things like P/SATA will not be adequate for future large drives.
Yeah but what's the point of storage if you can't access it? Or if you can, if it takes you 1 year to read the whole storage?

revolution wrote:
But that was my point actually. Glad to see you agree with me. Once the large storage comes in, NTFS will die.
Hmm, well that would not be surprising if you put it that way. All apps which used a hard disk/ram coding will die that way anyway.

Then again it's not like switching to a new filesystem is gonna be a problem. Switching to a different architecture is what I "fear" (unless there's an emulator or something) because it will break apps. Sad

_________________
Previously known as The_Grey_Beast
Post 03 Apr 2009, 23:15
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
Yeah but what's the point of storage if you can't access it? Or if you can, if it takes you 1 year to read the whole storage?
Why not? The usage model can change. No need to stick to the current model of SMs being limited size. No need to dig your toes in and ignore large SM just because they are (according to you) too big. There is no such thing as too big, right?!
Borsuc wrote:
Hmm, well that would not be surprising if you put it that way. All apps which used a hard disk/ram coding will die that way anyway.
But apps don't (or at least shouldn't) use HDD coding, that is a job for the OS. Just replace the memory, SM and FS driver code and no need to change your app.
Post 03 Apr 2009, 23:25
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
Why not? The usage model can change. No need to stick to the current model of SMs being limited size. No need to dig your toes in and ignore large SM just because they are (according to you) too big. There is no such thing as too big, right?!
What would you do?
You can't have a super-ultra-mega-hyper-resolution movies for example. They have a bandwidth and would exceed it if we don't increase the speed but ONLY the storage. So that goes out the window.

And it was one of the few advocates of such storage. (this includes anything real-time btw).

The only purpose it would have would be for archiving...

revolution wrote:
But apps don't (or at least shouldn't) use HDD coding, that is a job for the OS. Just replace the memory, SM and FS driver code and no need to change your app.
I was talking on a completely different change in the architecture, where there is no RAM, for example. (everything read directly from "HDD"). This would require change.

Obviously the "HDD" won't be a hard disk but something fast enough.

_________________
Previously known as The_Grey_Beast
Post 04 Apr 2009, 19:28
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
What would you do?
You can't have a super-ultra-mega-hyper-resolution movies for example. They have a bandwidth and would exceed it if we don't increase the speed but ONLY the storage. So that goes out the window.
Okay, good, now you are shifting the discussion towards what to do with the large storage. I presume this means you have accepted that it will happen.

So to address the problem of what to do with it is a lot harder to answer. Already in this thread I have made some wild speculations about possible usage of large storage, but these are not really meant as predictions. Usage models are really hard to predict but one thing I do expect is that if/when the storage comes then the users will find a use for it. Things that are currently being restricted by the current 'small' storage availability will suddenly become viable. I don't know what those things are, but here is a prediction (although a vague one at best): when the large storage comes, usage will change and people will find a way to make good use of it.
Post 05 Apr 2009, 00:49
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
I was talking on a completely different change in the architecture, where there is no RAM, for example. (everything read directly from "HDD"). This would require change.
Well, yes, that was what I also was trying to say actually. If future storage media (SM) becomes like RAM today then the idea is still valid. Just replace the OS drivers and the app won't know the difference. That is why we have OSes, to abstract all the gory details away from the apps.
Post 05 Apr 2009, 12:28
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
Okay, good, now you are shifting the discussion towards what to do with the large storage. I presume this means you have accepted that it will happen.
In the world of consumerism where the market decides whether a product exists or not, even if it is possible "theoretically", practically the market decides. Like I said it's hard to make use of big storage but small bandwidth. Except for archiving, but that will not be necessary for average joe users anyway.

I mean theoretically... you could even make a "CD" with manual "nano" holes... but what are you going to do with it if there's no reader? (suppose you ignored that light has diffraction, for example). Clearly you'll have to go with something else instead (like Flash Wink). The point I'm making is that theoretically storage already 'exists' since atoms are there already. What good is it if you can't read it in time?

CDs are still very common despite the fact that they are less efficient and even "pollute" the atmosphere more than let's say, DVDs... with .mp3 it's the same thing, since there are already "better" formats out there, but they aren't as popular.

Average Joe is not interested in archiving, cause then, tape backup would have become in previous years (when it was still efficient) a lot more popular.

The death of NTFS thus might be because it is stopped being supported by MS, because even if there are high-storage-but-"normal"-bandwidth available, the average joe is not gonna use it. For him, 16 EB is 'enough' even if he might have a Zetabyte storage.

revolution wrote:
Well, yes, that was what I also was trying to say actually. If future storage media (SM) becomes like RAM today then the idea is still valid. Just replace the OS drivers and the app won't know the difference. That is why we have OSes, to abstract all the gory details away from the apps.
But why would said OS then come up with those drivers since new apps probably won't be using them anyway? I guess it'll fall under emulation category Smile

_________________
Previously known as The_Grey_Beast
Post 05 Apr 2009, 14:34
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
In the world of consumerism where the market decides whether a product exists or not, even if it is possible "theoretically", practically the market decides. Like I said it's hard to make use of big storage but small bandwidth. Except for archiving, but that will not be necessary for average joe users anyway.
Have you forgotten to consider marketing? Mr Average Joe responds to good marketing. If some company has a good product and badly wants to sell it, then they "create the market" with good marketing. Have you ever heard the story about the two shoe salesmen that went to India and examined the prospects of selling shoes? The first salesman reported back: "Oh, boss, they don't wear shoes in India, there is no chance to sell anything there". The second salesman reported back: "Boss, I am so excited, they don't wear shoes in India yet, we can sell them by the millions".

Also, why are you assuming the bandwidth will be small? That is not a given fact. What do you consider small anyway? If the SM had a DDR3 interface would that be considered a small bandwidth?
Borsuc wrote:
But why would said OS then come up with those drivers since new apps probably won't be using them anyway? I guess it'll fall under emulation category Smile
The drivers abstract the details away, the app won't "know" that it is accessing a different type of storage medium, it will just think it is opening a file as usual and won't (and shouldn't) care about how and where it is stored.
Post 05 Apr 2009, 14:55
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
revolution wrote:
Also, why are you assuming the bandwidth will be small? That is not a given fact. What do you consider small anyway? If the SM had a DDR3 interface would that be considered a small bandwidth?
1 year = 365 days = 8760 hours = 525,600 minutes = 31,536,000 seconds

So to read 16 EB in a year we'll need about 544 GB/s.

revolution wrote:
The drivers abstract the details away, the app won't "know" that it is accessing a different type of storage medium, it will just think it is opening a file as usual and won't (and shouldn't) care about how and where it is stored.
Well functions like ReadFile (for windows) read the stuff into memory. Such functions would be worthless if the SM acted as RAM too, would they not? Wink

_________________
Previously known as The_Grey_Beast
Post 06 Apr 2009, 16:39
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Borsuc wrote:
revolution wrote:
The drivers abstract the details away, the app won't "know" that it is accessing a different type of storage medium, it will just think it is opening a file as usual and won't (and shouldn't) care about how and where it is stored.
Well functions like ReadFile (for windows) read the stuff into memory. Such functions would be worthless if the SM acted as RAM too, would they not? Wink
This could probably be handled by some smart memory mapping techniques... but it would be complicated code Smile

_________________
Image - carpe noctem
Post 06 Apr 2009, 16:43
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17450
Location: In your JS exploiting you and your system
revolution
Borsuc wrote:
So to read 16 EB in a year we'll need about 544 GB/s.
We are only discussing 1PB. You can't suddenly reaise the requirement by 16000 times, that is not a fair way to argue your case.
Borsuc wrote:
Well functions like ReadFile (for windows) read the stuff into memory. Such functions would be worthless if the SM acted as RAM too, would they not? Wink
This function for a driver is trivial. A simple memory mapping algo, just like RAM disk. Why do you think this is an unsolvable problem? It is a simple thing to do.
Post 06 Apr 2009, 23:03
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
I'm not sure but I doubt we'll get there with serial interfaces since they impose a limit on clock speed. Parallel ones have problems as well, synchronization problems to be more precise (why do you think IDE is obsolete nowadays?).

And of course, 1 year is way too much, but just for kicks. "Today's" harddisks can be accessed fully in around 3 days (the biggest ones) with 50 MB/s or so... (I know SATA supports up to a few gigs/s (4 GB/s IIRC) but that's theoretical, not counting HDD slowness).

Imagine doing a "bad sector scanning" to find out it takes 1 year to complete Laughing

As for the driver, well I'm not that experienced with OS internals so probably you're right Smile

_________________
Previously known as The_Grey_Beast
Post 07 Apr 2009, 01:25
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3 ... 5, 6, 7 ... 10, 11, 12  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.