Shared data store for server?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Shared data store for server?

grampajohn
Administrator
During a conversation today with Carsten, I realized that the piece that was really missing from the server architecture (in my opinion, and I think Carsten agrees) is another integration pattern called "shared database". In MinneTAC we call that piece the "Repository". It need not be a relational database; in fact, it's not clear we need the overhead of a relational database, and if we are strict about who gets to write to each element of the database (each table, in RDBMS terms), then transactions are not needed. The form of a shared repository is TBD, but the result will be that message traffic would be for communicating events, not bulk data, and associated data would in most cases be available for retrieval from the shared data store. This will eliminate the need, for example, for each component to keep its own list of all the current tariffs.

If you have your copy of Evans "Domain-Driven Design" handy, it's what he calls a "Repository" also - Chapter 6.

It's quite possible that this will all work with Hibernate, but I do not see any advantage to ORM, unless there's a need for SQL. In fact, if we don't need transactions, then the remaining question is whether we need persistence. We could use something like ObjectDB, which gives both, or we could just write a simple data access API that supports the shared types (like tariffs) and offers a few predefined queries, backed by an array or hashtable. That's how the MinneTAC repository works.

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

ddauer
I agree that having a (possibly restricted) shared database so that modules do not need to keep track of their own list of objects is a good idea.

In my opinion however, the main and probably only purpose of the database should be allowing modules to access historic data rather than requiring them to manually gather data based on an event. Message traffic right now includes data we assume to be necessary for the module to immediately process the event. This may also include "bulk data" I guess, but I'm not sure what you classify as "bulk data". I would definitely keep it the way it currently works because within the messaging infrastructure, the message data can be used to route or filter messages based upon certain criteria, among other things.

Also, the advantages (easy to configure and very easy to use, persistence, allowing module developers to use SQL) of using Hibernate definitely outweigh doing some manual programming.

David



On Wed, Dec 8, 2010 at 10:14 PM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
During a conversation today with Carsten, I realized that the piece that was really missing from the server architecture (in my opinion, and I think Carsten agrees) is another integration pattern called "shared database". In MinneTAC we call that piece the "Repository". It need not be a relational database; in fact, it's not clear we need the overhead of a relational database, and if we are strict about who gets to write to each element of the database (each table, in RDBMS terms), then transactions are not needed. The form of a shared repository is TBD, but the result will be that message traffic would be for communicating events, not bulk data, and associated data would in most cases be available for retrieval from the shared data store. This will eliminate the need, for example, for each component to keep its own list of all the current tariffs.

If you have your copy of Evans "Domain-Driven Design" handy, it's what he calls a "Repository" also - Chapter 6.

It's quite possible that this will all work with Hibernate, but I do not see any advantage to ORM, unless there's a need for SQL. In fact, if we don't need transactions, then the remaining question is whether we need persistence. We could use something like ObjectDB, which gives both, or we could just write a simple data access API that supports the shared types (like tariffs) and offers a few predefined queries, backed by an array or hashtable. That's how the MinneTAC repository works.

John


View message @ http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2053808.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

grampajohn
Administrator
On 12/08/2010 05:55 PM, ddauer [via Power TAC Developers] wrote:

> I agree that having a (possibly restricted) shared database so that
> modules do not need to keep track of their own list of objects is a good
> idea.
>
> In my opinion however, the main and probably only purpose of the
> database should be allowing modules to access historic data rather than
> requiring them to manually gather data based on an event. Message
> traffic right now includes data we assume to be necessary for the module
> to immediately process the event. This may also include "bulk data" I
> guess, but I'm not sure what you classify as "bulk data". I would
> definitely keep it the way it currently works because within the
> messaging infrastructure, the message data can be used to route or
> filter messages based upon certain criteria, among other things.

To me, sending around the entire list of current tariffs to every
customer model and every broker multiple times per day counts as bulk
data. The result is that every customer model has to start over again
every time figuring out which ones they might be interested in. It just
makes no sense to me.

>
> Also, the advantages (easy to configure and very easy to use,
> persistence, allowing module developers to use SQL) of using Hibernate
> definitely outweigh doing some manual programming.

Easy to configure and use if your objects are simple isolated objects.
Not necessarily so easy if your "objects" are really graphs or trees.

The disadvantage is having to do joins to reconstruct object graphs
(like tariffs). The relational model is just not a good match for
non-rectangular data. Here's a sample of the problem:
http://www.mojavelinux.com/blog/archives/2006/06/hibernate_get_out_of_my_pojo/.

John


>
> David
>
>
>
> On Wed, Dec 8, 2010 at 10:14 PM, grampajohn [via Power TAC Developers]
> <[hidden email] </user/SendEmail.jtp?type=node&node=2054632&i=0>> wrote:
>
>     During a conversation today with Carsten, I realized that the piece
>     that was really missing from the server architecture (in my opinion,
>     and I think Carsten agrees) is another integration pattern called
>     "shared database". In MinneTAC we call that piece the "Repository".
>     It need not be a relational database; in fact, it's not clear we
>     need the overhead of a relational database, and if we are strict
>     about who gets to write to each element of the database (each table,
>     in RDBMS terms), then transactions are not needed. The form of a
>     shared repository is TBD, but the result will be that message
>     traffic would be for communicating events, not bulk data, and
>     associated data would in most cases be available for retrieval from
>     the shared data store. This will eliminate the need, for example,
>     for each component to keep its own list of all the current tariffs.
>
>     If you have your copy of Evans "Domain-Driven Design" handy, it's
>     what he calls a "Repository" also - Chapter 6.
>
>     It's quite possible that this will all work with Hibernate, but I do
>     not see any advantage to ORM, unless there's a need for SQL. In
>     fact, if we don't need transactions, then the remaining question is
>     whether we need persistence. We could use something like ObjectDB,
>     which gives both, or we could just write a simple data access API
>     that supports the shared types (like tariffs) and offers a few
>     predefined queries, backed by an array or hashtable. That's how the
>     MinneTAC repository works.
>
>     John
>
>     ------------------------------------------------------------------------
>     View message @
>     http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2053808.html
>     <http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2053808.html?by-user=t>
>     To start a new topic under Power TAC Developers, email [hidden
>     email] </user/SendEmail.jtp?type=node&node=2054632&i=1>
>     To unsubscribe from Power TAC Developers, click here
>     <
>
>
>
>
>
> ------------------------------------------------------------------------
> View message @
>
http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2054632.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

grampajohn
Administrator
In reply to this post by ddauer
On 12/08/2010 05:55 PM, ddauer [via Power TAC Developers] wrote:
> I agree that having a (possibly restricted) shared database so that
> modules do not need to keep track of their own list of objects is a good
> idea.

What do you think of neo4j (neo4j.org)? It's a "NoSQL" graph database.
No joins, much better for handling objects and relationships. See
http://blog.neo4j.org/2010/10/spring-data-and-neo4j.html for info on
their collaboration with Spring.

>
> In my opinion however, the main and probably only purpose of the
> database should be allowing modules to access historic data rather than
> requiring them to manually gather data based on an event. Message
> traffic right now includes data we assume to be necessary for the module
> to immediately process the event. This may also include "bulk data" I
> guess, but I'm not sure what you classify as "bulk data". I would
> definitely keep it the way it currently works because within the
> messaging infrastructure, the message data can be used to route or
> filter messages based upon certain criteria, among other things.
>
> Also, the advantages (easy to configure and very easy to use,
> persistence, allowing module developers to use SQL) of using Hibernate
> definitely outweigh doing some manual programming.
>
> David
>
>
>
> On Wed, Dec 8, 2010 at 10:14 PM, grampajohn [via Power TAC Developers]
> <[hidden email] </user/SendEmail.jtp?type=node&node=2054632&i=0>> wrote:
>
>     During a conversation today with Carsten, I realized that the piece
>     that was really missing from the server architecture (in my opinion,
>     and I think Carsten agrees) is another integration pattern called
>     "shared database". In MinneTAC we call that piece the "Repository".
>     It need not be a relational database; in fact, it's not clear we
>     need the overhead of a relational database, and if we are strict
>     about who gets to write to each element of the database (each table,
>     in RDBMS terms), then transactions are not needed. The form of a
>     shared repository is TBD, but the result will be that message
>     traffic would be for communicating events, not bulk data, and
>     associated data would in most cases be available for retrieval from
>     the shared data store. This will eliminate the need, for example,
>     for each component to keep its own list of all the current tariffs.
>
>     If you have your copy of Evans "Domain-Driven Design" handy, it's
>     what he calls a "Repository" also - Chapter 6.
>
>     It's quite possible that this will all work with Hibernate, but I do
>     not see any advantage to ORM, unless there's a need for SQL. In
>     fact, if we don't need transactions, then the remaining question is
>     whether we need persistence. We could use something like ObjectDB,
>     which gives both, or we could just write a simple data access API
>     that supports the shared types (like tariffs) and offers a few
>     predefined queries, backed by an array or hashtable. That's how the
>     MinneTAC repository works.
>
>     John
>
>     ------------------------------------------------------------------------
>     View message @
>     http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2053808.html
>     <http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2053808.html?by-user=t>
>     To start a new topic under Power TAC Developers, email [hidden
>     email] </user/SendEmail.jtp?type=node&node=2054632&i=1>
>     To unsubscribe from Power TAC Developers, click here
>     <
>
>
>
>
>
> ------------------------------------------------------------------------
> View message @
>
http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2054632.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

ddauer
In reply to this post by grampajohn


On Thu, Dec 9, 2010 at 1:15 AM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
On 12/08/2010 05:55 PM, ddauer [via Power TAC Developers] wrote:

> I agree that having a (possibly restricted) shared database so that
> modules do not need to keep track of their own list of objects is a good
> idea.
>
> In my opinion however, the main and probably only purpose of the
> database should be allowing modules to access historic data rather than
> requiring them to manually gather data based on an event. Message
> traffic right now includes data we assume to be necessary for the module
> to immediately process the event. This may also include "bulk data" I
> guess, but I'm not sure what you classify as "bulk data". I would
> definitely keep it the way it currently works because within the
> messaging infrastructure, the message data can be used to route or
> filter messages based upon certain criteria, among other things.
To me, sending around the entire list of current tariffs to every
customer model and every broker multiple times per day counts as bulk
data. The result is that every customer model has to start over again
every time figuring out which ones they might be interested in. It just
makes no sense to me.

Ok you're right. We'll built it in a way that large lists will be stored somewhere and not be sent with the message. Spring also provides something called "Claim Checks" (1) which we can hook into the architecture anytime should we see performance issues resulting from larger messages.

- David

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

grampajohn
Administrator
I've read through the Claim Check section, and I think this could work.
It's a nice solution.

John


On 12/09/10 08:05, ddauer [via Power TAC Developers] wrote:

>
>
> On Thu, Dec 9, 2010 at 1:15 AM, grampajohn [via Power TAC Developers]
> <[hidden email] </user/SendEmail.jtp?type=node&node=2057827&i=0>> wrote:
>
>     On 12/08/2010 05:55 PM, ddauer [via Power TAC Developers] wrote:
>
>     > I agree that having a (possibly restricted) shared database so that
>     > modules do not need to keep track of their own list of objects is
>     a good
>     > idea.
>     >
>     > In my opinion however, the main and probably only purpose of the
>     > database should be allowing modules to access historic data rather
>     than
>     > requiring them to manually gather data based on an event. Message
>     > traffic right now includes data we assume to be necessary for the
>     module
>     > to immediately process the event. This may also include "bulk data" I
>     > guess, but I'm not sure what you classify as "bulk data". I would
>     > definitely keep it the way it currently works because within the
>     > messaging infrastructure, the message data can be used to route or
>     > filter messages based upon certain criteria, among other things.
>     To me, sending around the entire list of current tariffs to every
>     customer model and every broker multiple times per day counts as bulk
>     data. The result is that every customer model has to start over again
>     every time figuring out which ones they might be interested in. It just
>     makes no sense to me.
>
>
> Ok you're right. We'll built it in a way that large lists will be stored
> somewhere and not be sent with the message. Spring also provides
> something called "Claim Checks" (1) which we can hook into the
> architecture anytime should we see performance issues resulting from
> larger messages.
>
> - David
>
> (1)
> http://static.springsource.org/spring-integration/reference/htmlsingle/#new-claim-check
>
>
> ------------------------------------------------------------------------
> View message @
> http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2057827.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

grampajohn
Administrator
In reply to this post by grampajohn
I've been doing some research, and I withdraw my objection to using Hibernate/SQL. The conditions under which a graph database like neo4j wins big involve much deeper graphs than anything we will see. I'm thinking we will likely need to store Tariffs and Subscriptions in a way that allows flexible queries.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Shared data store for server?

ddauer
Ok great, so we'll try setting things up with Hibernate.

- David

On Sat, Dec 11, 2010 at 5:30 PM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
I've been doing some research, and I withdraw my objection to using Hibernate/SQL. The conditions under which a graph database like neo4j wins big involve much deeper graphs than anything we will see. I'm thinking we will likely need to store Tariffs and Subscriptions in a way that allows flexible queries.


View message @ http://power-tac-developers.975333.n3.nabble.com/Shared-data-store-for-server-tp2053808p2069392.html

To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Loading...