Re: Output to text files or a database?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

kd wrote:
I was reading The Art of Unix Programming yesterday, and was inspired
to rework and split up an large monolithic application of mine.

This software has to do with the retrieval and processing of travel
information (flights, hotels, etc) from different sources. Each source
has travel data formatted differently.

With plain text files being the universally loved file format, I
planned on writing small apps that would pull down data from these
different sources, and write them to text files in a universal format.

Another small app would read these text files, generate sql insert
statements from the data, and dump all the data into a relational
database.

I proposed this to a coworker, and he asked why the text files were
even necessary -- why not just immediately generate sql statements from
the different formats, and use the tables in the DB as the universal
format? Textual representations of that data could just be generated
from the database, after all.

Is he right? Is there any reason to use text files as an intermediate
format?

I don't know. But I'd have some questions, like

- - what do you do if the database server isn't available when the app
needs to store it's data?

- - what do you do if the database server isn't available when the app
that needs the data requests it?

- - does the data need to be preprocessed, hand edited, or otherwise
subjected to non-db processing?

- - database apps are largish. Are you getting into the same sort of
maintenance issue that you are moving out of, by building one larger
app rather than two smaller apps?

- - is there any reason to want to perform ad-hoc analysis on the text
files? can you do the same against the database? How easily?

- - is there any reason to want to perform alternate processing on the
data (does it or can it serve any other purpose other than what you
intend)? can you do this processing on text files? how easily, as
compared to against a database?

- - what about recoverability? Auditability? Are these important?

My gut is telling me to keep them, but my gut isn't good at
convincing people. :) What am I missing?

- --

Lew Pitcher, IT Specialist, Corporate Technology Solutions,
Enterprise Technology Solutions, TD Bank Financial Group

(Opinions expressed here are my own, not my employer's)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEBdeJagVFX4UWr64RAkx+AKDi3OvUbWHOrnOAGPjoE2N9glC77QCgzp5y
ww+bkaEz2x2/bq7CqRfiog0=
=muEF
-----END PGP SIGNATURE-----
.

Relevant Pages

  • Re: Output to text files or a database?
    ... This software has to do with the retrieval and processing of travel ... With plain text files being the universally loved file format, ... from the database, after all. ... My gut is telling me to keep them, but my gut isn't good at ...
    (comp.unix.programmer)
  • C# creation of SQL CE databases
    ... It is to use British date format: ... I am having problems with the comparison of dates between dates entered in the app and the database. ... I can format the dates in C# and then format dates using convert in SQL and compare the two.. ... If it's not possible to set the format for the whole database is it possible to specify the format in a create table statement or to add some type of constraint to ensure the date is formatted correctly. ...
    (microsoft.public.dotnet.framework.compactframework)
  • Date/Region In SQL CE
    ... It is to use British date format: ... I am having problems with the comparison of dates between dates entered in the app and the database. ... I can format the dates in C# and then format dates using convert in SQL and compare the two.. ... If it's not possible to set the format for the whole database is it possible to specify the format in a create table statement or to add some type of constraint to ensure the date is formatted correctly. ...
    (microsoft.public.sqlserver.ce)
  • Re: Portable Database Choice
    ... I searched this group quite a bit looking for database alternatives and did find the options below from this search. ... I'm posting this in the hope it can be of use to other developers in a position similar to mine where I needed a low cost alternative to Pocket Access. ... One app requires synchronization between desktop and mobile device, the other requires a push of data from the desktop to mobile. ...
    (microsoft.public.dotnet.framework.compactframework)
  • Re: Portable Database Choice
    ... > database alternatives and did find the options below from this search. ... One app requires ... > push of data from the desktop to mobile. ... > Both of these apps used Pocket Access on the device with Peter Foot's ...
    (microsoft.public.dotnet.framework.compactframework)