flat assembler
Message board for the users of flat assembler.

Index > Heap > multi-GB files, tools to analyze and 'grep' data?

Author
Thread Post new topic Reply to topic
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
Actually it sounds like I've answered my question in the same sentence,
but the problem isn't that simple. I know Linux has grep, but I'm under
Windows.

Note: this is not piracy - all the channels are free here in Estonia and they
are for my private use. I just can't watch all the channels in parallel on 31st Smile

Details:
~70GB *.ts file with 6 parallel (A+V) streams of DVB-T channels with EPG.
I understand the fileformat (http://en.wikipedia.org/wiki/MPEG_transport_stream)
but what would be the simplest way to
1) split the ts into 6 different files (6 TV-channels) - VLC doesn't help here, strips EPG
2) get the EPG-info and use it to slice the file further on show borders

Example:
Recording takes place at 18:00 for 10h (~2MB/s meaning ~70GB)
EPG says "show one starts at 19:30, lasts 0:50"
show0.ts - dump from 18:00...19:30
show1.ts - dump from 19:30...20:20
show2.ts - etc.

Actually it doesn't matter if the shows overlap. It would just take a huge
burden off doing everything manually - which I'm willing to do if there's no
better solution

thanks Wink
Can I do it in FASM quickly? PHP Razz ?

_________________
My updated idol Very Happy http://www.agner.org/optimize/


Last edited by Madis731 on 30 Dec 2009, 14:36; edited 1 time in total
Post 30 Dec 2009, 13:54
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17247
Location: In your JS exploiting you and your system
revolution
Just record everything first and then post process later when you are ready to watch.

Or is it that you want to watch all 6 channels simultaneously? Confused
Post 30 Dec 2009, 14:05
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
I suppose "mencoder" should be able to splice streams as you want (but I never tried it). Not sure about EPG info.
Post 30 Dec 2009, 14:15
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
@vid: VLC and mencoder are both able to do some of the work, but then I want to process further (slice on different TV-show borders). I think there is no magic bullet for that. I stared at MSDN for a while but I think I will program something later... tomorrow Smile

@revolution: I am recording it all - because I can't watch them simultaneously, but the post-processing will be horror.


I can't seem to find FILE_APPEND_DATA equivalent in FASM (maybe its the fault of win64a.inc?)

EDIT: Found something else, I'll see if it can do the trick - http://www.tsreader.com/tsreader/index.html - or at least ease my following work.

I would need to go Pro if I wanted to use the features Razz
TSReader wrote:
TSReader Professional's archiving mode. In the sample, two channels from a mux are being recorded with file splitting based on EPG data. A modest modern PC with five DVB-T cards, some large hard drives and plenty of memory can record everything from a service like Freeview in the UK -- simultaneously.
Post 30 Dec 2009, 14:35
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
I'm in a pickle (or maybe I'm a pickle Smile ). The TSReader (any version) is almost perfect, but VLC can also dump TS-stream. I have a 46GB stream now.

TSReader can read a file, but it doesn't understand that I need to record something from that file that happened a few days back - not now. So EPG doesn't live up to its name.

The problem is that SMPlayer/Mplayer/VLC will play it fine, but they don't have a sense of time. VLC can seek to any position percentage-wise, but it doesn't show any time. *Mplayer's (also mencoder written by the same group) won't allow me to go to any position percentage/byte/time-wise, but it will allow me to start from the beginning and scroll through the *seven-hour* file by 10sec or 1min increments.

VDub (VeeDub?) will not play anything besides MPEG-1 / PCM avi Sad

Smile How could I filter out these clips?
2:28:59-3:01:30
5:02:38-5:57:24
from a 7:01:38 video

I also converted them to MBs:
16123,22-19642,23 (MB)
32751,47-38678,41 (MB)
file is 47838450296 bytes in size.
Post 03 Jan 2010, 17:54
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
DustWolf



Joined: 26 Jan 2006
Posts: 373
Location: Ljubljana, Slovenia
DustWolf
To be guessing, looking up MPEG-TS on wikipedia, the stream is transmitted in packets, which can act as standalone:
Quote:
A packet is the basic unit of data in a transport stream. It consists of a sync byte, whose value is 0x47, followed by three one-bit flags and a 13-bit Packet Identifier (PID). This is followed by a 4-bit continuity counter. Additional optional transport fields, as signaled in the optional adaptation field, may follow. The rest of the packet consists of payload. Packets are 188 bytes in length[4] , but the communication medium may add some error correction bytes to the packet. DVB-T/C/S uses 204 bytes and ATSC 8-VSB, 208 bytes as the size of emission packets (transport stream packet + FEC data). ATSC transmission adds 20 bytes of Reed-Solomon forward error correction to create a packet that is 208 bytes long [5]. The 188-byte packet size was originally chosen for compatibility with ATM systems [6][7] .


If the file is a file pretending to be a stream, rather than an actual stream, you will also have an index at the start (not necessarily at the start). You can read the index to figure out which packets you need to copy out, or you can extrapolate it. The final file will also need an index so you either have to write one, or use a tool that can do it for you (a newer version of VLC should be able to do that).

Since you say your players can't scroll the file, I would assume the index must be missing in the source file.

Good enough?

EDIT: Apparently, TS doesn't have an index, the index is in fact only in AVI files and follow a Microsoft specification. The point that TS files are not scrollable does not come as much of a shock thus, since a TS is supposed to be nothing but a live stream. You can probably extrapolate the packet positions. To add scrolling capability, you could encapsulate your TS in an AVI format and add an index then.

LP,
Jure
Post 03 Jan 2010, 18:04
View user's profile Send private message AIM Address Yahoo Messenger MSN Messenger Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.