2012 tournament status

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

2012 tournament status

grampajohn
Administrator
To participants in the 2012 tournament -

I have been tracking down the cause of the huge numbers of HourlyCharge instances that are recorded in the server logs. This is causing performance problems when multiple brokers offer multiple variable-rate tariffs, but should be fixable on the server in a transparent way (see issue #620). There are also some errors in the way some brokers are specifying variable rates, but those do not seem to be causing problems.

I should be able to have a tested fix available by the end of today, my time. In the meantime, the games can continue to run, but some will get slow and we may need to cut some short if they are not making progress or using too much disk space.

In the future, we may need to be much more annoying about getting your help to run test tournaments with your brokers. This is a scale issue that we unfortunately have not had the resources to test independently.

Thanks much -

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

gbuijs
John is still testing the fix, so we're pausing the games.

When John is happy with the result, we'll resume the games at 07:00 GMT.
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

grampajohn
Administrator
Dear participants -

The latest 0.6.3-SNAPSHOT seems to solve the problem of huge logfiles when variable rates are used. We've seen HourlyCharge instances arriving without the corresponding Rates, but this should not cause a problem for the server. It may not have the desired effect for the broker that is sending them, of course.

If you want to use this server locally, you will need to update powertac-server and use the updated server-distribution pom.xml, or you can of course just update all the sources and run from the server-main or visualizer projects in Eclipse. Note that because this fix was in the common module, we had to update version dependencies in all the other server modules. You do not need to update your brokers; the change is backward-compatible, and only affects logging behavior. There are no functional changes.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

grampajohn
Administrator
Dear participants -

I feel like I'm in the twilight zone right now. On Monday, we were confident that we could run a tournament without major issues. Once we got past some login problems, we were in good shape until we found the logfiles exploding with variable-rate updates (HourlyCharge instances and related bits). The fix for that went slightly awry, and I had that problem solved 6 hours ago. Ever since then, I have been unable to run tests outside of my IDE. Inside Eclipse, everything works. Outside, I keep seeing the version from last night instead of the version that's in eclipse. That's true even when I clear out the local repo, check to see that the source is correct, then clean and re-install or re-deploy using maven. I am getting the same behavior on multiple machines.

In any case, until we figure out why maven cannot see the new code, we cannot run games with Aston or Soton - as far as I can tell, those are the only agents using variable rates.

Wolf wants to re-schedule the whole tournament, but my feeling is that if we cannot run with real agents, we cannot know whether we are ready or not. I believe that once we get this weird maven problem figured out, we should be good to go, but of course something else could crop up that's hidden behind this problem. My hope is that you will stick with us. We can re-run games that were incomplete, and we should be able to run the complete tournament and get good results. We just cannot finish by tomorrow.

Please respond with your suggestions on how to proceed. I see no reason why we cannot finish this tournament given a few more days, but we may want to pause it over the weekend.

Thanks again.

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

achryso
The Mertacor Team agrees with you. If you have fixed the problem there is no need to not continue the tournament as is. Moving it would only cause more frustration, since no real testing will be done in the process.

It would be nice to know as soon as possible when the games will begin or / and the whole tournament schedule from now on.

Maybe the two broker games that do not include the two teams that are presently using variable tariffs can be scheduled and completed in the near future.

Keep up the good work and may the maven God help you soon.

We will keep our brokers running just in case.

Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

icmoon
In reply to this post by grampajohn
Dear John and Wolf:

Thanks for your effort that prepared this tournament.

KAIST team would like to have this tournament to be continued after your fix.

Again, thanks for your great effort on this.

Il-Chul Moon

Assistant Professor
Dept. of Industrial and Systems Engineering
KAIST



On Thu, Sep 27, 2012 at 5:00 AM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
Dear participants -

I feel like I'm in the twilight zone right now. On Monday, we were confident that we could run a tournament without major issues. Once we got past some login problems, we were in good shape until we found the logfiles exploding with variable-rate updates (HourlyCharge instances and related bits). The fix for that went slightly awry, and I had that problem solved 6 hours ago. Ever since then, I have been unable to run tests outside of my IDE. Inside Eclipse, everything works. Outside, I keep seeing the version from last night instead of the version that's in eclipse. That's true even when I clear out the local repo, check to see that the source is correct, then clean and re-install or re-deploy using maven. I am getting the same behavior on multiple machines.

In any case, until we figure out why maven cannot see the new code, we cannot run games with Aston or Soton - as far as I can tell, those are the only agents using variable rates.

Wolf wants to re-schedule the whole tournament, but my feeling is that if we cannot run with real agents, we cannot know whether we are ready or not. I believe that once we get this weird maven problem figured out, we should be good to go, but of course something else could crop up that's hidden behind this problem. My hope is that you will stick with us. We can re-run games that were incomplete, and we should be able to run the complete tournament and get good results. We just cannot finish by tomorrow.

Please respond with your suggestions on how to proceed. I see no reason why we cannot finish this tournament given a few more days, but we may want to pause it over the weekend.

Thanks again.

John



If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/2012-tournament-status-tp4025126p4025129.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

jurica.babic
In reply to this post by grampajohn
Dear John,

CrocodileAgent team is willing to continue with the tournament.

We think it is good to have huge variety of brokers participating because this is the only right way to evaluate game design as well as software implementation of PowerTAC.

If there is anything we can contribute to, please let us know!

Cheers,
Jurica
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

gbuijs
It looks John has won the battle with Maven.
His fix is working perfectly now.

I will resume the tournament in the morning.
I hope to start it at 07:00 GMT, but it's rather late here so don't be mad if it's half an hour later.
I will check if everybody's broker is checking in before resuming.

Cheers,
Govert
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

grampajohn
Administrator
gbuijs wrote
It looks John has won the battle with Maven.
His fix is working perfectly now.
For future reference - I had a transitive dependency that was pulling in an older version of common. That dependency appears to have been caused by the build dependencies in server-interface and genco, which both use aspects defined in common and must be woven at build time. So re-deploying those two modules fixed the problem.

It's fairly subtle, and not at all intuitive, at least to me.

Game 81 between SotonPower and Mertacor is running right now if you want to look at it. It seems to be handling the variable rates from SotonPower correctly, without exploding the logs.

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

grampajohn
Administrator
In reply to this post by gbuijs
Dear colleagues -

Game 81 finished normally, and seems complete and correct. The server handled the variable rates from SotonPower just fine. But it ran a bit slower than we would like - it took about 3.5 hr instead of the nominal 2 hr. I did some research to figure out where the time is going, and it seems to be dominated by the time spent sending messages to brokers (looks like about 25 msec/message). This was a 2-player game, so we can expect that larger games will be slower yet.

I have my JMS expert looking into the problem, and if anyone out there knows JMS well enough to have an opinion about performance in Spring I would be happy to discuss it. If we come up with a significant improvement before the tournament is finished we'll see about doing an update, which should not affect brokers. But this should not deter us from proceeding with the tournament. Just be aware that games will run a bit slowly for now.

I know JMS is used in very high-performance applications, and I think it's reasonable to expect message times to be well under 5 msec. We just need to improve our understanding of how to use it efficiently.

All the best -

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

HarryRose
Hi,

The SotonPower team are happy for this competition to run as a pilot, but we feel that there needs to be a lot of testing with agents before reliable, publishable results can be obtained from the server.  It needs to be thoroughly tested such that it can be confirmed the server works to spec.

It would be good to schedule regular unofficial friendly matches that run over weekends to which agents can join as they like.  This will allow us to test our agents as well as for you to test the server.  It might also build people's confidence in PowerTAC, encouraging more teams to participate.  These friendlies could run year-round until an official competition in aamas/aaai etc.

We're also happy to send you a jar file with our agent so you can test the server offline.

Harry
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

grampajohn
Administrator
Dear colleagues -

Several games ran successfully today, although they were painfully slow. One game, #7, is currently blocked at timeslot 1760, 11 timeslots from the end, and is not making progress. It appears to be getting connection exceptions. There are no other exceptions, and the logfile sizes are reasonable.

So here's the status as I see it.

1. The server and visualizer seem to be working reliably.
2. The agents, other than BanTex, appear to be working, and there have not been login problems like we experienced in June.
3. There is a performance problem related to sending JMS messages, especially with the larger games. It will not be a trivial fix, but there is hope for a 1-2 day fix that would not affect brokers. A more comprehensive fix might require some message re-packaging, and that would definitely affect brokers.
4. There is some evidence that the factored-customer model is not evaluating variable-rate tariffs as well as it should. I don't know the details yet, and I don't know whether it's out-of-spec or not. Probably the spec is not sufficiently detailed in this area.

So the question is, what should we do? Obviously we all want data that is robust and publishable. It would be nice to have some usable data in the next couple of weeks, because the AAMAS deadline is approaching. The AAAI deadline is in January, if I'm not mistaken. I see two possibilities, not necessarily exclusive:

1. Take a few days, perhaps until next Tuesday 10/2, and try for a fix to the performance problem and perhaps an improvement in the factored customer model. Then run a set of perhaps 50-60 games with agents that are definitely going to participate. Unless I hear back from the BanTex people, I'll assume it's 6 agents, and we'll probably do game sizes of 2,4, and 6.

2. Give you some time to look at the data we have and make suggestions about how it could be improved, do some updates to the logging, and run another series later, possibly in early to mid-November. This might require some small broker changes, although ideally it would only touch the common and core modules.

In the meantime, it would indeed be helpful to have some of your agents available for running tests. Harry has offered; one or two others would be helpful.

Please let us know how you would like to proceed. Thanks again for all your hard work.

John
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

achryso
The Mertacor Team would be happy to help you by giving her current agent, which seems to work ok in order to run some more tests.

But as a matter of fact I will run more tests myself to stress test the models, but more eyes are better than 2...

I don't have problem with any of the proposed solutions, even though I am more sceptic with short-time-to-be tournament as for how good data it can provide for papers and deductions over the game, in my humble opinion.
Reply | Threaded
Open this post in threaded view
|

Re: 2012 tournament status

HarryRose
The SotonPower team believe that option two (postpone until November) is the best option.  Given the issues that have been discovered throughout this competition, I don't think the results from this tournament are reliable enough for use in a publication.  

We're of the opinion that PowerTAC 2013 should be the next official competition, with the competition in November being a pilot. The November results can then be used to encourage participation in 2013.
Reply | Threaded
Open this post in threaded view
|

RE: 2012 tournament status

Wolf
Administrator
The suggestion by the SotonPower team sounds reasonable. I`m chairing the IEEE Smart Grid conference (www.sg-tep.org  -- paper deadline is Oct 31st) in Nuremberg on Dec 3 and 4. We could run another pilot then or if the community wants a real competition, but my feeling is that it should be a last pilot. I`ll discuss with John and Govert over the next days and let you know.

Thanks much for all your hard work and patience. This is a very innovative and complex competition, and a good community around it is invaluable. The Power TAC team really appreciates your participation, and we think, once these issues are solved we will be able to run successful competitions in the future and get valuable and deep research results out of it.

We`ll get back to you about this soon.

Thanks again,

Wolf

From: HarryRose [via Power TAC Developers] [ml-node+[hidden email]]
Sent: 28 September 2012 09:30
To: Wolf Ketter
Subject: Re: 2012 tournament status

The SotonPower team believe that option two (postpone until November) is the best option.  Given the issues that have been discovered throughout this competition, I don't think the results from this tournament are reliable enough for use in a publication.  

We're of the opinion that PowerTAC 2013 should be the next official competition, with the competition in November being a pilot. The November results can then be used to encourage participation in 2013.


If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/2012-tournament-status-tp4025126p4025143.html
To start a new topic under Power TAC Developers, email ml-node+[hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

-------------------------------------------------------------------------------------------disclaimer-------------------------------------------------------------------------
De informatie verzonden in dit e-mail bericht inclusief de bijlage(n) is vertrouwelijk en is uitsluitend bestemd voor de geadresseerde van dit bericht. Lees verder: http://www.eur.nl/email-disclaimer
The information in this e-mail message is confidential and may be legally privileged. Read more: http://www.eur.nl/english/email-disclaimer
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Reply | Threaded
Open this post in threaded view
|

RE: 2012 tournament status

AstonTAC team
We are happy with your decision to postpone the pilot/competition to November or beginning of December.

We will be grateful if you can publish some more details of the customer model for the new server as soon as possible. Especially, we don't understand what is going on with the customer behaviour. It seems that the new server does not respond to our very low tariff price. We offered 10% of the wholesale market average price. However, we only managed to get less than 5% of the customer subscription. It is not normal. While the sever before the competition could deal with our tariff properly. We think there are some problems with the way how customers evaluate the tariff in general.

thanks

AstonTAC