Filtering the needed facts of very large internal database files
Posted: 10 Jun 2022 17:38
I have in an internal database USA’s custom statistics of monthly trade in goods with different countries. In the database are now 17.5 million facts. Every month new facts are added to it.
The problem in creating pages and charts of different product group’s import and export (for example Gases Sitc 34) with different countries, using and going through the large database in memory is very time consuming. Producing the yearly and monthly charts and html-pages of nearly 100 product groups did take almost 24 hours. So developing and/or correcting the pages was rather “difficult”.
Finally I divided the database in smaller each one product groups facts having files. Reading the rows of such large data file, which is fast, and writing the wanted fact rows in a new file is easy. With the new internal database files the time to make the charts and pages can now be made in some tens of minutes.
Is it possible to create the needed internal database asserting the chosen rows during this reading the rows of the input stream?
The problem in creating pages and charts of different product group’s import and export (for example Gases Sitc 34) with different countries, using and going through the large database in memory is very time consuming. Producing the yearly and monthly charts and html-pages of nearly 100 product groups did take almost 24 hours. So developing and/or correcting the pages was rather “difficult”.
Finally I divided the database in smaller each one product groups facts having files. Reading the rows of such large data file, which is fast, and writing the wanted fact rows in a new file is easy. With the new internal database files the time to make the charts and pages can now be made in some tens of minutes.
Is it possible to create the needed internal database asserting the chosen rows during this reading the rows of the input stream?
Code: Select all
clauses
readProductTerms(InFile,ProductCode):-
Input = inputStream_file::openFile8(InFile),
readProductStream(Input,string::format(",\"%\",",ProductCode)),
Input:close(),!.
predicates
readProductStream : (inputStream Input,string SearchStr) procedure(i,i).
clauses
readProductStream(Input,_):-
Input:endOfStream(),
!.
readProductStream(Input,SearchStr):-
String = Input:readLine(),
handleString(String,SearchStr),
readProductStream(Input,SearchStr).
predicates
handleString:(string RowsString,string SearchString).
clauses
handleString(String,SearchStr):-
LEN = string::length(String),
LEN > 8,
string::search(String,SearchStr)=N,
N>0,!,
?????
assert this fact in the string to internal database in memory
.