Results 1 to 9 of 9

Thread: Looking for a suggestion for "dynamic" tables

Threaded View

  1. #1

    Thread Starter
    Super Moderator FunkyDexter's Avatar
    Join Date
    Apr 2005
    Location
    An obscure body in the SK system. The inhabitants call it Earth
    Posts
    7,957

    Looking for a suggestion for "dynamic" tables

    Bit of background:-
    I'm implementing a system where a user can specify multiple sources of entities (in this case people) and then define some matching criteria that can be used to identify common entities across sources. E.g. Joe Bloggs in Source A is the same person as Joe Bloggs source B because they have the same forename and surname.

    (matching criteria will actually be far more complex than that - I use Levenshtein distancing to calculate similarity scores on each field then a weighted average to generate an overall score and assign a match based on an overall threshold being reached)

    I cannot know the sources a user will have at design time.

    So my basic approach is to have a UI where the user can define a source, it's properties and it's matching criteria to another source. That can then create the necessary supporting tables - basically a table per source with a field per property (call this a source data table). The user will then be responsible for populating the source tables and my matching algorithm will run over the resultant data.

    The problem:-
    How to create those source data tables. At present I'm using dynamic sql to generate the necessary create table statements but you can't put metadata (table name, column name etc.) in a sql parameters. That means I'm having to use concatenation and that, in turn, risks sql injection. That's what I'd like to get past. Can anyone suggest a way of sanitizing the inputs for this.

    I should say that this app will be used by trusted users who would have direct DB access anyway so injection risks aren't the worst thing in the world for me but it still feels shoddy.

    Failing that, I could consider a redesign of the matching part of the system. Two obvious approaches would be a NoSQL database or an single Entity Attribute Value Pair table. I don't like either of these though. It would make it difficult for the users to populate the data - they're usually decent sql users but they're not going to know a NoSql platform and asking them to populate a EAV design would be a nightmare. (Aside, for anyone discovering it for the first time, I do NOT recommend EAV - it seems like a good idea at first, you WILL pay for it later.)

    I also considered a single table for all sources with the combined field set for all sources but that feels pretty ugly too. It would have a massive number of empty fields on any given row.

    I'm open to suggestions for a redesign but it needs to be easily populatable to a moderately able but non professional sql user.


    So, any thoughts?
    Last edited by FunkyDexter; Aug 7th, 2025 at 04:32 AM.
    The best argument against democracy is a five minute conversation with the average voter - Winston Churchill

    Hadoop actually sounds more like the way they greet each other in Yorkshire - Inferrd

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width