Server restructuring is complete

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Server restructuring is complete

grampajohn
Administrator
Dear colleagues -

This evening I have pulled over all the pieces of the server restructuring. There are two major changes:

First, the server is now a standard multi-module maven project. That means, for example, that if you run mvn compile or mvn test in the powertac-server directory, it will build and test the whole server. It also means that virtually all of the common dependencies are factored out into the top-level pom.xml. Most of the modules now have very simple poms, because they inherit from powertac-server. Equally important, setting up Eclipse (and other IDEs, I'm sure) is now quick and painless. All you do is import powertac-server as an "existing maven project" and everything gets set up, including all the plugin modules. Note that common is not a server module; it is separate, because it's also needed by brokers. You don't even need to install it in eclipse unless you want to work on it - it gets downloaded like any other dependency, from our maven repo at Minnesota.

Second, in order to make the server buildable using our Jenkins CI server, the server is also structured as a multi-module git project. That means that git knows about the directory relationships that maven wants, which is to have all the child modules inside the directory containing the parent pom. Once you have everything set up, you can still switch branches in any of the modules, but there are some caveats. The first-time setup process is explained in the README.md in powertac-server. Once you have that set up, you can just get rid of the old module directories that are beside powertac-server (except common) - they will not be used. If you work on powertac-server, you may notice that git status shows one or more of your modules as changed. These changes arise from updated versions of your modules, and should not be committed unless you definitely know what you are doing. It might make sense to add them to the .gitignore file to prevent this.

I hope this is the last major disruption for server developers. I think once you have updated your environment and found where everything got moved to, you will find it much easier to work, because STS and maven both have the same ideas about how everything is put together.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
Nguyen has loaded up the new server and shared some experiences:

Nguyen wrote
I just checked out the powertac-server, ran your script, import it into STS. It loaded so smooth that i didn't have a chance to complain.

When i tried to run the PowerTacServer class as Java app from STS, i got the error message "Launch Error: Editor does not contain a main type".  It seems that STS doesn't like/know the maven structure.  Do we need to do anything to run server within STS?
and then the next morning:
I closed my STS window last night after seeing the mentioned error.  I noticed that the src folder at not regconized by STS as "source folder" and thought about trying to get STS to do that.  This morning, i opened STS up and somehow all those source folders are now regconized by STS as "source folder" and i could run server from STS now.  I'm back in business :)
I think the key is that the project configuration in STS is derived from the pom.xml files, and STS does not always figure out everything during the initial load. You may need to do (right-button on powertac-server) Maven->Update project configuration, and let it do all the subprojects.

To run the server, create a run configuration, project server-main, class org.powertac.server.PowerTacServer, with a single argument ../src/main/resources/bootstrap.txt.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
In reply to this post by grampajohn
I have updated the Getting Started page to reflect the new server structure. I would appreciate feedback or edits if you try it out and find something missing or confusing.

Thanks in advance.

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete - user master branch

grampajohn
Administrator
Before anyone gets confused, in the powertac-server project on github, we are now developing on the master branch, rather than spring-master. The old grails version is now on grails-master, and master is again the "default" branch.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

Prashant Reddy
In reply to this post by grampajohn
Hi John,

Quick question... how do I change the log4j config?

The file in powertac-server/src/main/resources/log.config and the
equivalent files in all the submodules only contain appenders for
"test" state and trace -- where are the powertac0 logs coming from?

I tried adding a console appender to the above log.config files but
that doesn't seem to do anything and I can't find a different source
of log4j config.

Thanks much,
Prashant
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

chris.flath
This may be related - I keep receiving the following warning:

log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
In reply to this post by Prashant Reddy
Prashant Reddy wrote
Quick question... how do I change the log4j config?
The runtime log config is created programmatically in server-main/.../server/LogService. If you can see a better way to do this, I would be very interested.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
In reply to this post by chris.flath
chris.flath wrote
This may be related - I keep receiving the following warning:

log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
This has always been happening, because the log config is apparently not accessible early enough in the startup process. I have not taken the time to figure out why. I would welcome a solution, or a new issue about it if you don't have a solution, but for now I think it is a low-priority issue.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete - installation on Windows

grampajohn
Administrator
This afternoon I worked with Zoheb to get the new server set up on his Windows/STS 2.8 environment. He was having two problems. First, git was complaining about a missing dll, and did not get the accounting and auctioneer directories created. But when we did git submodule init, they got created (along with another complaint about a missing dll). Second, for some reason maven running inside STS was unable to resolve some of its own plugin dependencies, and so it could not build anything. We ended up going into a term window and building with maven (mvn clean compile inside powertac-server). Once we had done that, it seemed to take two rounds of maven->update project configuration before it was happy.

I hope this helps. I know next to nothing about Windows, so if anyone wants to add their knowledge to this discussion, please do.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

Prashant Reddy
In reply to this post by grampajohn
No, loading from LogService is fine... I just didn't think to look in
the code.  But, I'm curious why is it not loaded from a file -- is
this test config versus prod config issue (I don't see a switch for
that in the code if that is indeed the issue)?

This new setup works beautifully by the way -- I struggled with the
STS setup for over a day last week whereas this new setup took less
than an hour!  Thanks!

Prashant
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

achryso
In reply to this post by grampajohn
Usually for the trick was done when I moved or copied the log.config file in my test directory of each project. That was on the older versions before maven submodule configuration took over.

I guess the same can be applied now in the current modules.

Does this make sense?



Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
On 11/17/2011 04:26 AM, achryso [via Power TAC Developers] wrote:
> Usually for the trick was done when I moved or copied the log.config
> file in my test directory of each project. That was on the older
> versions before maven submodule configuration took over.
>
> I guess the same can be applied now in the current modules.
>
> Does this make sense?

I am not sure I understand what problem this solves, but the log.config
files are in src/test/resources in each module.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

achryso
I was referring to this problem stated by Prashant and Chris

This may be related - I keep receiving the following warning:

log4j:WARN No appenders could be found for logger (org.springframework.context.support.ClassPathXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Reply | Threaded
Open this post in threaded view
|

Re: Server restructuring is complete

grampajohn
Administrator
Yes, I see this message every time I run tests or run the server. It seems that Spring is trying to use the logger before we have it set up. It's not a critical issue, but if you have a solution we should fix it.

John
Reply | Threaded
Open this post in threaded view
|

Need windows developer to adapt git module script

grampajohn
Administrator
In reply to this post by grampajohn
In powertac-server/src/main/scripts there is a shell script 'submodule-setup.sh' that pulls in current versions of all the git submodules. It would be quite helpful if someone with more knowledge of Windows than I have could create a Windows version of that. I assume the existing script works on Macs, or I would have heard about it.

Any volunteers?

Thanks in advance.

John
Reply | Threaded
Open this post in threaded view
|

Re: Need windows developer to adapt git module script

chris.flath
Your script runs fine if placed the powertac-server directory and called from the windows git console.
Reply | Threaded
Open this post in threaded view
|

Re: Need windows developer to adapt git module script

nhartlie
Right, you just have to copy it to the head folder. Do you - Windows User - also get some error messages regarding some missing dll files when executing the script? Anyway it's workin'.
Reply | Threaded
Open this post in threaded view
|

Re: Need windows developer to adapt git module script

chris.flath
I did not encounter missing dll problems. Maybe some Windows git hickup?
Reply | Threaded
Open this post in threaded view
|

Re: Need windows developer to adapt git module script

nhartlie
Maybe... But as it works, I don't really care...
Reply | Threaded
Open this post in threaded view
|

Re: Need windows developer to adapt git module script

grampajohn
Administrator
In reply to this post by chris.flath
chris.flath wrote
Your script runs fine if placed the powertac-server directory and called from the windows git console.
Thanks, Chris. Much easier than I expected. I have updated the getting-started page with this advice. Could you please take a look and see if it's accurate? If not, feel free to fix it.

Thanks a million!

John