Web and database development and servers should become one system

I have to admit it but I am the kind of programmer (or Software Writer, thank you DHH) who is more about getting a job done, rather than producing code that adheres to whatever today's pattern of choice is, or writing code that will be easy for someone else to comprehend (or even myself a month later).

In my current position, and in most of my positions, I have often been the sole developer responsible for some new projects and modifications to existing applications, and time is usually critical. Had I ever worked in a larger team I probably would have picked up some better habits.

I mention this because it influences what I want and need from development environments and server systems.



Recently I've been researching and experimenting with alternate methods to various things involving development, particularly database access, and have come to the conclusion that the systems that we have evolved to today are not ideal.

Looking at database access from a web perspective, let's look at the some of what we have that I think isn't ideal.

Firstly, we have a web server application (and hardware) and a database server application (and probably separate hardware). Two separate systems that need to have a connection and move data backwards and forth. Although I see the obvious advantage of the systems being separate in that each can do what it does best independently of the other, I wonder if it wouldn't be more efficient to have the functionality of these two in one application (on one or more servers).

Then from a development perspective, we create our database, create our website and then we either have to just code in SQL queries which is prone to errors, or use some sort of ORM so our web application can learn the structure of the database. To be efficient we do things like make sure our select queries return only the data needed so we can minimise data transfer between systems. For some complex operations we might create a stored procedure so as to minimise back and forth between these two systems. Now we are coding in two different development environments and this could open up issues with the development process, for example different skill requirements, deployment issues, and source control.

If the database was part of the same development environment we could directly access the database structure from our code.

If the database was part of the web server we could access any data needed without fear of inefficiency from just moving data.

In the "old days" I used to create some fairly complex information system applications using a DBMS called dBase (I used versions III and IV). This was a database first, but with a programming language for controlling how data is displayed and entered. Although these were desktop applications it's not a dissimilar situation to a database-driven website or application. Although you specified which databases you wanted to use, there was no connections between systems, and the programming language had database query and manipulation as a (the!) core feature.

I imagine a conceptually similar language for my ideal combined web and database system. Don't get me wrong, I don't want the dBase IV programming language. It wasn't a great system in hindsight, as every "table" was a database of its own and from memory there was no relationships.

I use Linq to SQL primarily in my day job and I think that this is a good example of how SQL-like queries could be integrated into a programming language, but I propose the language to be even more SQL-like. With L2S we often have to jump through a few hoops to get the resulting SQL query we want, which means that to be a good and efficient L2S developer, you have to be an expert in L2S and an expert in SQL. Again, why two systems?

If I were to improve on L2S I would better handle different types of database joins, and in my combined system concept, there would be no need to specify columns to be returned: just access any column from any specified table or joined table.

Doesn't it drive you crazy when you add some new functionality to an application and you need a different column set than you have needed elsewhere so you have to then create a new database view and model, or (gasp) return an anonymous type?

I've mentioned these thoughts in online discussion before and have almost universally been shot down, but I still see merit in these ideas. Personally, I think that although most developers welcome new technologies and techniques in general, that many are so set in their ways that they can't embrace something that goes against some of the things that the separation and pattern police have brainwashed them with.

All of these thoughts lead me to check out if there was actually something that did something like I was dreaming, and someone pointed me to Ruby on Rails which I had previously bought a book on but I think I lost interest after finding half the book relating to what appeared to be a very long winded process on how to install what's needed on Windows (was an early version, so has probably changed).

Although since I've never even done "Hello World!" in Rails, I have read a bit about it and watched a few online videos showing its features, and it looks very appealing in many ways.

I'm not so sure about going back to the command line, and I don't know if I like the idea of having so much CRUD code created automatically, but I really should give it a try and see rather than prejudging.

Also, having been a Microsoft developer for so long, I do love Visual Studio and all that goes with it like Intellisense, code completion, formatting, and debugging. I keep reading that a lot of this isn't needed in RoR but maybe I have too many bad memories of writing Classic ASP in Notepad.

I really like that in VS when I write some code with L2S that, being strongly typed, it tells me immediately when I get something wrong. I watched a video with one of the RoR developers showing some code and I wasn't impressed by having to save code, open in browser, check some sort of web server real time logging to see where things went wrong.

Also in my book on Rails, there is a bit of hand-coded SQL queries to do some of the things that can't be done automatically.

So as much as I'd like to try producing something in RoR, I don't think it's the answer to my prayers.

I don't have the details for how this would all work, I'm just thinking out loud on the concept, and hoping to stir up a bit of discussion and maybe someone reading sees some merit in the idea and develops the next big thing in web servers and development.