Database work has been mildly successful. Used insert but that ended up being very slow, instead I ended up using load data infile and it is A LOT quicker. Using inserts it was running around 40mins to insert around 300k records into the db. When I ended up writing the results from the news server out to disk and then using load data infile I loaded the same 300k records in about 3 minutes.
Now I am trying to figure out how I am going to order the rest of the tables and what information they should contain. I think that the raw tables may not actually be needed in the end.
Right now I am thinking;
Release filename table. (This will have parts etc, will need to check for existance
Release table (Releases, size and date).
This is all TBC….