Is all this complexity really necessary?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Is all this complexity really necessary?

grampajohn
Administrator
After downloading the server and all the stuff the server seems to depend on, I have 221 Mb of stuff, I have no idea what most of it is for, and it won't build. I feel like we are several meters deep in gratuitous complexity. Could someone please explain how all this stuff is going to make our lives easier? Will you be able to explain it to a researcher who wants to add a model without having to become an expert in multiple specialized enterprise Java frameworks?

As you can tell, I'm feeling frustrated, and worrying that our project might be going off the rails. Or is that grails?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is all this complexity really necessary?

Carsten Block
Administrator
John -

at the moment we're doing the "heavy lifting" behind the scenes here in Karlsruhe to get the server up and running and as I mentioned on our last conf call: We're not yet there. This is also the very reason why we did not release any new version of it yet and we did not encourage anybody to check out the code and try it out. And as we're really trying to focus on getting everything right in the server code we didn't have sufficient time to write very comprehensive "getting started" documentation yet. It's on our list but it's not priority one. 

The idea in behind the whole new server core is that an interstested developer who wants to contribute for example a particular tariff rule enforcer module, will only have to focus on the respective module and not at the server as a whole. He will be provided with a template, which, in this particular case, consists of one single class (see below)  that he might want to extend in whatever way he prefers in order to implement his logic. Particularly for customer modules, we will also provide a template module that show how to build the logic using RePast. 


Sample class of a tariff rule enforcer. 

public class TariffRuleEnforcerImpl implements TariffRuleEnforcer {

    @Override
    public boolean accept(TariffReplyCommand tariffReplyCommand) {
        // Default implementation simply returns true
        return true;
    }
}


As you can tell, I'm feeling frustrated, and worrying that our project might be going off the rails.
I'm sorry about that, John. I suggest that you give us some more time so that we can work on the server (and its documentation), while you could finalize the specification for the tariff model together with Prashant and Chris? (Chris is sending out the protocol from Friday meeting on Tuesday after his return from a short travel).

Or is that grails? 
No. No grails in the server anymore. At the moment not even groovy code. It's all pure Java.

Cheers,
Carsten



Am 05.12.2010 um 05:33 schrieb grampajohn [via Power TAC Developers]:

After downloading the server and all the stuff the server seems to depend on, I have 221 Mb of stuff, I have no idea what most of it is for, and it won't build. I feel like we are several meters deep in gratuitous complexity. Could someone please explain how all this stuff is going to make our lives easier? Will you be able to explain it to a researcher who wants to add a model without having to become an expert in multiple specialized enterprise Java frameworks?

As you can tell, I'm feeling frustrated, and worrying that our project might be going off the rails. Or is that grails?


View message @ http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2020969.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is all this complexity really necessary?

grampajohn
Administrator
Thanks, Carsten. I spene much of the afternoon and evening yesterday
trying to get set up to understand the server, and it was an exercise in
frustration. I am accustomed to having a fairly deep knowledge of the
inner workings of projects I work on, and this one so far is quite
opaque to me. It's a very uncomfortable feeling. I wish I had more time
to dig into all the reading material you have suggested, but that just
is not going to happen soon. One thing that would help greatly is a
short 2-3 sentence explanation of what all these pieces are and why they
are important.

Cheers -

John


On 12/05/2010 05:18 AM, Carsten Block [via Power TAC Developers] wrote:

> John -
>
> at the moment we're doing the "heavy lifting" behind the scenes here in
> Karlsruhe to get the server up and running and as I mentioned on our
> last conf call: We're not yet there. This is also the very reason why we
> did not release any new version of it yet and we did not encourage
> anybody to check out the code and try it out. And as we're really trying
> to focus on getting everything right in the server code we didn't have
> sufficient time to write very comprehensive "getting started"
> documentation yet. It's on our list but it's not priority one.
>
> The idea in behind the whole new server core is that an interstested
> developer who wants to contribute for example a particular tariff rule
> enforcer module, will only have to focus on the respective module and
> not at the server as a whole. He will be provided with a template,
> which, in this particular case, consists of one single class (see below)
> that he might want to extend in whatever way he prefers in order to
> implement his logic. Particularly for customer modules, we will also
> provide a template module that show how to build the logic using RePast.
>
>
> Sample class of a tariff rule enforcer.
>
> public class TariffRuleEnforcerImpl implements TariffRuleEnforcer {
>
> @Override
> public boolean accept(TariffReplyCommand tariffReplyCommand) {
> // Default implementation simply returns true
> return true;
> }
> }
>
>
>> As you can tell, I'm feeling frustrated, and worrying that our project
>> might be going off the rails.
> I'm sorry about that, John. I suggest that you give us some more time so
> that we can work on the server (and its documentation), while you could
> finalize the specification for the tariff model together with Prashant
> and Chris? (Chris is sending out the protocol from Friday meeting on
> Tuesday after his return from a short travel).
>
>> Or is that grails?
> No. No grails in the server anymore. At the moment not even groovy code.
> It's all pure Java.
>
> Cheers,
> Carsten
>
>
>
> Am 05.12.2010 um 05:33 schrieb grampajohn [via Power TAC Developers]:
>
>> After downloading the server and all the stuff the server seems to
>> depend on, I have 221 Mb of stuff, I have no idea what most of it is
>> for, and it won't build. I feel like we are several meters deep in
>> gratuitous complexity. Could someone please explain how all this stuff
>> is going to make our lives easier? Will you be able to explain it to a
>> researcher who wants to add a model without having to become an expert
>> in multiple specialized enterprise Java frameworks?
>>
>> As you can tell, I'm feeling frustrated, and worrying that our project
>> might be going off the rails. Or is that grails?
>>
>> ------------------------------------------------------------------------
>> View message @
>> http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2020969.html
>> <http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2020969.html?by-user=t>
>> To start a new topic under Power TAC Developers, email [hidden email]
>> </user/SendEmail.jtp?type=node&node=2021890&i=0>
>> To unsubscribe from Power TAC Developers, click here
>> <
>
>
>
>
> ------------------------------------------------------------------------
> View message @
>
http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2021890.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

AW: Is all this complexity really necessary?

Daniel Schnurr
In reply to this post by grampajohn
AW: Is all this complexity really necessary?

Just a comment on the complexity of the architecture:

The vision of the comprising content within the Power TAC scenario was always quite complex and extensive.
And complexity rather grew over the last year than decreased due to new ideas and additional information. (At least until two weeks ago).
Therefore we agreed that a modular architecture would be the key to handle this complexity in the design but also to allow for parallel work at different places.

In my opinion the current pipes-and-filters/message-driven/EAI-pattern-based approach allows us precisly to handle these issues.
There are technical challengeges in "heavy-lifting" of this environement, of course. But if we want to build a platform that can face the mentioned requirements we will not find a solution that prevent us from some complexity at a certain point. (Or as Gregory likes to put it: "There is no free lunch").
Through the decoupled entities and the flexible connection between them, the current approach gives us a framework to manage this in a reasonable way.
Not only in technical ways but it is also very powerful in abstracting, separating and encapsualting the design and understanding of the server logic.

Another argument in discussions was always that the system should support a project with a long-time horizon. With Spring/Spring Integration we use state of the art technology that exactly should ensure this. (And by the way it is one of the reasons why I personally am still motivated after the technology change and re-design).

We had a technology in place that reduced the technical complexity by a certain degree. This was Grails. It automated quite a lot of the things we do now manually.
The consensus in the end was that this automation might provide too much over-head and that we dig deeper to maintain control. We are now at a point where we begin to know how we can use that control to generate real benefit. I don't see the point about thinking going back/changing technology again at this point.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: AW: Is all this complexity really necessary?

grampajohn
Administrator
Daniel and all -

I understand quite a bit about building highly modularized and
configurable applications, and the approach you have taken is one of
many possible approaches. The pipe-and-filter / message-driven / EAI
scheme has advantages and disadvantages; many of the advantages are
primarily realized in highly-scalable systems where modules written by
different groups for different purposes can be deployed freely in a
distributed environment and hooked together as needed to solve business
problems. I would not have chosen this approach for our system, but I
think it can work. The last time we discussed server design, we were
going to use Spring, and we were not going to use OSGi modules. We were
also going to build with Ivy, not Maven. Now we have Spring Integration,
Virgo, Roo, and Maven. With the exception of Maven, I don't know what
any of those things are, I don't know what role they are playing in the
design, and so far I've been unsuccessful in my attempts to set up a
development environment.

As co-leader of this project (with Wolf), I find this situation very
frustrating and fairly scary. I also have four students here at
Minnesota who are ready to contribute, and are stuck (after spending
considerable time trying to understand what you are doing) because they
cannot get started. I have spent most of this weekend trying to figure
it out myself, and I've run out of time.

I am not suggesting that we tear everything apart and start over. But we
have a number of talented people committed to do their part to realize
our vision who cannot participate effectively right now. It's critical
that they become productive soon - within the next week - because we are
up against a very aggressive deadline.

John


On 12/05/2010 05:32 PM, Schnurr [via Power TAC Developers] wrote:

> Just a comment on the complexity of the architecture:
>
> The vision of the comprising content within the Power TAC scenario was
> always quite complex and extensive.
> And complexity rather grew over the last year than decreased due to new
> ideas and additional information. (At least until two weeks ago).
> Therefore we agreed that a modular architecture would be the key to
> handle this complexity in the design but also to allow for parallel work
> at different places.
>
> In my opinion the current
> pipes-and-filters/message-driven/EAI-pattern-based approach allows us
> precisly to handle these issues.
> There are technical challengeges in "heavy-lifting" of this
> environement, of course. But if we want to build a platform that can
> face the mentioned requirements we will not find a solution that prevent
> us from some complexity at a certain point. (Or as Gregory likes to put
> it: "There is no free lunch").
> Through the decoupled entities and the flexible connection between them,
> the current approach gives us a framework to manage this in a reasonable
> way.
> Not only in technical ways but it is also very powerful in abstracting,
> separating and encapsualting the design and understanding of the server
> logic.
>
> Another argument in discussions was always that the system should
> support a project with a long-time horizon. With Spring/Spring
> Integration we use state of the art technology that exactly should
> ensure this. (And by the way it is one of the reasons why I personally
> am still motivated after the technology change and re-design).
>
> We had a technology in place that reduced the technical complexity by a
> certain degree. This was Grails. It automated quite a lot of the things
> we do now manually.
> The consensus in the end was that this automation might provide too much
> over-head and that we dig deeper to maintain control. We are now at a
> point where we begin to know how we can use that control to generate
> real benefit. I don't see the point about thinking going back/changing
> technology again at this point.
>
>
>
>
> ------------------------------------------------------------------------
> View message @
> http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2024557.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: AW: Is all this complexity really necessary?

Carsten Block
Administrator
John and all,

thanks for this discussion. There are lots of pros and cons for each technology choice. 
At this point I feel that you John, and I should go bilateral to clarify the issues raised. 

Today I'm sick at home with terrible headaches and a cold. Maybe not the best preconditions for such a discussion. Hopefully tomorrow is better. Can we put it in front or after the skype call with Wolf and Sebastian?

Thanks,
Carsten  



Am 06.12.2010 um 01:53 schrieb grampajohn [via Power TAC Developers]:

Daniel and all -

I understand quite a bit about building highly modularized and
configurable applications, and the approach you have taken is one of
many possible approaches. The pipe-and-filter / message-driven / EAI
scheme has advantages and disadvantages; many of the advantages are
primarily realized in highly-scalable systems where modules written by
different groups for different purposes can be deployed freely in a
distributed environment and hooked together as needed to solve business
problems. I would not have chosen this approach for our system, but I
think it can work. The last time we discussed server design, we were
going to use Spring, and we were not going to use OSGi modules. We were
also going to build with Ivy, not Maven. Now we have Spring Integration,
Virgo, Roo, and Maven. With the exception of Maven, I don't know what
any of those things are, I don't know what role they are playing in the
design, and so far I've been unsuccessful in my attempts to set up a
development environment.

As co-leader of this project (with Wolf), I find this situation very
frustrating and fairly scary. I also have four students here at
Minnesota who are ready to contribute, and are stuck (after spending
considerable time trying to understand what you are doing) because they
cannot get started. I have spent most of this weekend trying to figure
it out myself, and I've run out of time.

I am not suggesting that we tear everything apart and start over. But we
have a number of talented people committed to do their part to realize
our vision who cannot participate effectively right now. It's critical
that they become productive soon - within the next week - because we are
up against a very aggressive deadline.

John


On 12/05/2010 05:32 PM, Schnurr [via Power TAC Developers] wrote:

> Just a comment on the complexity of the architecture:
>
> The vision of the comprising content within the Power TAC scenario was
> always quite complex and extensive.
> And complexity rather grew over the last year than decreased due to new
> ideas and additional information. (At least until two weeks ago).
> Therefore we agreed that a modular architecture would be the key to
> handle this complexity in the design but also to allow for parallel work
> at different places.
>
> In my opinion the current
> pipes-and-filters/message-driven/EAI-pattern-based approach allows us
> precisly to handle these issues.
> There are technical challengeges in "heavy-lifting" of this
> environement, of course. But if we want to build a platform that can
> face the mentioned requirements we will not find a solution that prevent
> us from some complexity at a certain point. (Or as Gregory likes to put
> it: "There is no free lunch").
> Through the decoupled entities and the flexible connection between them,
> the current approach gives us a framework to manage this in a reasonable
> way.
> Not only in technical ways but it is also very powerful in abstracting,
> separating and encapsualting the design and understanding of the server
> logic.
>
> Another argument in discussions was always that the system should
> support a project with a long-time horizon. With Spring/Spring
> Integration we use state of the art technology that exactly should
> ensure this. (And by the way it is one of the reasons why I personally
> am still motivated after the technology change and re-design).
>
> We had a technology in place that reduced the technical complexity by a
> certain degree. This was Grails. It automated quite a lot of the things
> we do now manually.
> The consensus in the end was that this automation might provide too much
> over-head and that we dig deeper to maintain control. We are now at a
> point where we begin to know how we can use that control to generate
> real benefit. I don't see the point about thinking going back/changing
> technology again at this point.
>
>
>
>
> ------------------------------------------------------------------------
> View message @
> http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2024557.html
> To start a new topic under Power TAC Developers, email
> <a href="x-msg://12/user/SendEmail.jtp?type=node&amp;node=2024775&amp;i=0" target="_top" rel="nofollow">[hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>



View message @ http://power-tac-developers.975333.n3.nabble.com/Is-all-this-complexity-really-necessary-tp2020969p2024775.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: AW: Is all this complexity really necessary?

grampajohn
Administrator
On 12/07/2010 02:38 AM, Carsten Block [via Power TAC Developers] wrote:
> Today I'm sick at home with terrible headaches and a cold. Maybe not the
> best preconditions for such a discussion. Hopefully tomorrow is better.
> Can we put it in front or after the skype call with Wolf and Sebastian?

Carsten and I had a very good conversation this morning (my time), and I
feel much more comfortable. I am convinced that the installation can be
completely automated, so eventually (soon?), developers will need to
only download a small shell from powertac.org and run a script, which
will retrieve the necessary pieces and wire them together. We will also
institute daily builds at both Karlsruhe and Minnesota, and the KIT
group will generate the documentation at least once/day and post it with
a link in the server wiki. Also linked into the server wiki will be the
most recent test reports.

The other piece of our conversation today was about the need for a
shared data store. I'll address that in another message.

Cheers -

John
Loading...