NoSQL Database to support Key Value Pairs

[FunkyDexter]

Hi All

For a long I've been intrigued by the flexibility using a Key Value Pair aproach for data storage would give me. I've written various db apps over the years and had the constant frustration of restructuring the tables as the real world domain changes. How great would it be to put some of that power straight into the hands of the customer, so as they need new attributes added to e.g. their customers, they could just do it themselves? Sounds wonderful but anyone who's spent any time looking into implementing this aproach using a relational database will know that it's fraught with problems. I actually started implementing this pattern to see how I got on and, while I was able to overcome alot of the issues, I can personally vouch for the fact that it's a real pig and I'm pretty convinced that some of the problems aren't solluble at all.

Recently, though, I've been reading that NoSQL datastores are far better suited to this aproach so I think I'd like to try some out and see where they take me. I have absolutely no experience with NoSQL aproaches so don't really know where to start. There's quite a few out there and, realistically, I'm not going to have a chance to try them all out so I thought I'd solicit some recomendations first. Has anyone used any NoSQL database and what were their experiences.

A few considerations:-
This isn't for a commercial project, just little old me playing around, so I can't afford to be paying enterprise level costs. Ideally if there's a proper "free" aproach (similar to microsofts express products) that would be great, a free trial period followed by a managable cost would also work for me.
At heart I'm a .net guy. I can work in Java and C++ if needs be but, in an ideal world, I'd like something that integrates well with the .Net family of products.
Finally, it's the flexibility of KVP I'm after. "Big Data" considerations (which NoSQL is also touted for) aren't really on my radar for now.

That's where I'm at. Any suggestions?

[Serge]

I can tell you first hand about NoSQL database as I'm using it in conjunction with SQL Server. NoSQL is not a solution for ALL scenarios, there are pros and cons. I personally use MongoDB but there are others. You will have to assess if the pros outweighs the cons for you to use it. Great database solution, with auto-sharding and failover servers built-in. You can run it on commodity hardware. In terms of speed it kills ANY RDMS by a MILE. The good thing about MongoDB is it has a thriving community, they support many drivers for pretty much all programming languages (including .NET). Since you are a .NET developer, you can use their C# driver (this driver also has a LINQ support so your queries will be 90% similar to any other LINQ queries). Write speeds are ENORMOUS. I've tested the write speeds in MongoDB with C# and LINQ and I can do 10,000 writes per second on my development box (SATA 7.2k RPM drive). In production, MongoDB is installed on a SSD drive and it just screams. I'm using MongoDB for activity stream and user notifications. No matter what I throw at it, my queries run in <5 ms. In comparison with SQL SERVER if you have complex joins and million of records (despite having proper indexes), MongoDB is about 10 to 100 times faster (depending on data, indexes and complexity of your queries) and best of all, it runs on commodity hardware. In order for you to have SQL Server run at least somewhat comparable, you have to spend thousands (if not hundred of thousands) of dollars on servers. Also, SQL Server doesn't really scale horizontally (having more than 1 server to act as 1), although MS has a SQL Server Datacenter Edition which allows you to bundle many servers to act as one but the cost is pretty much unacceptable to a regular user.

Now the MongoDB cons:
MongoDB does not support transactions, so if you have a Insert/Update/Delete bulk operations and you need them either all to success or to fail, MongoDB cannot help you. So you have to weigh in all your requirements. If you require transactions, you most likely not be able to use it. I can't speak about other NoSQL databases (although I've researched them all when I needed a solution).

In my case I've chosen MongoDB because it had a strong support for .NET and can be installed pretty much on any OS (including Windows). Also for activity stream and notifications, I didn't require transactions, its pretty much fire and forget. You should try it out. MongoDB has a large community, good documentation and code examples for any language.

And the best of all - It's open-source and 100% FREE.

http://www.mongodb.org/

Here are some C# with LINQ examples:
http://docs.mongodb.org/ecosystem/tu...csharp-driver/

[FunkyDexter]

Hi Serge and thanks for the info.

I'd come across folks talking about Mongo in my own initial research and people seem to speak highly of it. Mongo and Virtuoso were probably the two I was looking most seriously at so getting some gen on one of them is ideal.

It's open-source and 100% FREE

That's major plus point, right there

Knowing I can talk to it from C#'s also a major plus. I don't really know linq but I've looked at it a few times and it's similar enough to SQL to make me confident I can pick it up quick. Horizontal scaling and speed are less of an issue to me but it's nice to know I won't have anything to worry about in that area.

The transaction issue is a little concerning but not too bad. If I think back over my career I can probably count the number of times I've really needed a transaction on one hand... OK, two hands and both feet but that's still not many in over 20 years. I can almost always achieve what I want in a single SQL statement but whether that will still be true in a NoSQL world remains to be seen. I've come across people using phrases like "eventually consistent" which I think is related to this. I.e. the whole "transaction" may not happen as one but it'll all get there eventually. I need to get my head round that a bit more. And I assume I can probably write some stuff that, when I really do need atomicity, can roll back half changes.

Overall it definitely sounds like one that's worth trying out.

Thanks.

[abhijit]

If you're foraying into LINQ, give LINQPad a shot. Its a nifty tool that I can't recommend enough.

[FunkyDexter]

That looks interesting. Anything that shallows the learning curve is good.