Finals 2017

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Finals 2017

gbuijs
Dear Participants,

Monday the 12th is the start of the finals.
We'll start at 09:00 UTC sharp, so make sure you have 2 agents running by then.
We won't have time to wait for latecomers and it would be a pity to miss games.

We'll have 2 occasions where we have to halt the tournament.
Due to network maintenance there will be some outage around these times:
- Monday the 12th, 17:30 UTC
- Monday the 19th, 17:30 UTC
We'll make sure no games are running at these times.

As always keep an eye on this mailing list, your email and IRC.

Good luck!

Cheers,

Govert
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
Dear colleagues -

You may have noticed that (at least) one of the brokers in the current competition is making money by running up huge positive imbalances. We believe the problem is an incorrect configuration for the cost slope in the balancing market in the case of positive imbalance (down-regulation). It's taking some time to work through the test cases and confirm that the problem is not deeper than that.

We hope to have a confirmed fix later today. The question is what to do with the games that have already run. Should we re-run all of them, and thereby extend the duration of this round? Alternatively, we might be able to reduce the number of full sets of 3, 5, and 8-broker games. Govert could let us know what both alternatives would do to the schedule.

We hope to have more definite news later today (local time at 93 degrees west).

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

gbuijs
At the moment we're running game 25 and 26 of 60 8-player games.
That took us roughly 25 hours.

We could proceed with the remaining 8, 5 and 3-player games after the fix.
The estimated endtime of the tournament is 12:00 - 16:00 (UTC) Wednesday the 21th.

I see 3 options:
1 Replay the first 26 games with the fixed configuration.
2 Remove the 26 games from the tournament results.
3 Include the 26 games, even though this skews the results.

I would prefer option 1, as this would result in better research data.
But this would require all participants to run their agents for an extra day.

Thoughts?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
Dear colleagues -

I found the problem in the balancing market, an extra minus sign when computing the p1 value (called p_imb in the spec) in the case of overall down-regulation. That led to brokers getting paid a positive amount for having a positive imbalance when the overall imbalance was also positive. That value should be positive only for a tiny positive imbalance, but should turn negative quickly. I believe the specification is correct.

I have re-computed and updated test cases, and deployed a new 1.4.4-SNAPSHOT server if you want to see what it does. In the meantime, our plan is to re-start the final round from the top starting tomorrow morning Rotterdam time. This gives you at least a brief chance to fix your broker in case you were relying on the incorrect sign, but of course if your broker was learning it on the fly it should adapt correctly.

I hope to have a 1.4.4 release out before the restart, so that the logs will be labeled 1.4.4 rather than 1.4.3 or 1.4.4-SNAPSHOT.

Thanks for your patience with an old geezer programmer. Please let us know as soon as possible if this plan is inconvenient for you.

John and Wolf


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

SPOT
Hi John,

Can you please give us the link to the 1.4.4-SNAPSHOT? We would like to test our broker before the finals. Are you going to resume the game at tomorrow 9 UTC?

Just a request, Is it possible to run at least a brief (e.g. 1 day) trial round with the new server before re-starting the tournament? This change could affect a lot of things about the market that would have been learned from historical data rather than online, so a trail period will be really helpful.  

Thanks
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
SPOT wrote
Can you please give us the link to the 1.4.4-SNAPSHOT?
It's on a branch server-distribution/release-1.4.3. With a little more reflection I probably could have come up with a better name...
 We would like to test our broker before the finals. Are you going to resume the game at tomorrow 9 UTC?

Just a request, Is it possible to run at least a brief (e.g. 1 day) trial round with the new server before re-starting the tournament? This change could affect a lot of things about the market that would have been learned from historical data rather than online, so a trail period will be really helpful.  
Those are questions for Govert and Wolf. You can certainly run local games right away.

Again, apologies for the disruption.

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

gbuijs
We'll run a trial tournament with 20 8-player games.
This will take +/- a day.

We'll start tomorrow (wednesday) 09:00 UTC.
To make sure all agents will participate, the trial will be called the same.
So there's no reason to adjust your broker.properties.

We'll continue the finals thursday 09:00 UTC, assuming all trial games have ended.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
In reply to this post by grampajohn
Dear colleagues -

Power TAC server version 1.4.4 is now available on Maven Central, so you should now be able to run the release. Release notes and a download link is on the participant's wiki.

Thanks again for your patience. We decided to act and rearrange the competition schedule in order to produce more meaningful results. Our hope is that this will make it much easier to justify the statistics in the papers I'm sure you will all be writing soon...

As always, please let us know if you have questions or problems with this.

Best of luck with the competition.

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

Ansel
In reply to this post by gbuijs
Hi John and Wolf,

We, the COLDPower Team, are developing a multiagent broker that try to learn how maximize gains and minimize pays for each market.  Mainly, We uses severals MDPs and reinforced learning that that try to discover how make money for each one.

Our current broker COLDPower takes advantage from the gains by positive imbalance and the retail actions depends of global cash position. With any change in the behavior of the simulation, we need to train again the agents that compose our broker.

Each year  we combine a retail market strategy with others.

Last year our broker learned to make money on wholesale market by buying cheap and selling expensive and then rules were changed. After that, we rebuild a new strategy and trains it in one week. Fortunately the new strategy worked.

 This Year we try to make money by combining wholesale market  orders with gains by positive imbalance and then the rules are changed too.  But now we will have 24 hours to rebuild a new strategy and trains it. Also as we assumed that in finals the code can not be improved, this week we involve the main part of the developers team in others projects and tasks at 600 miles from our lab.

We like the challenges, and it is a big one.

cheers

Ansel
COLD TEAM
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
Ansel and all -

When we saw the strange results in the balancing market, my first reaction was that your group was doing exactly what we hope broker developers would do, which is to use machine learning to explore the rather large parameter space in the simulation environment. I think the COLDPower team should get a special prize for that so effectively.

But after understanding exactly what was happening, our second thought was that you had thereby discovered a flaw, and that by exploiting it you would certainly win. However, the community, including your group, would then end up learning very little from the competition other than that the balancing market was not implemented correctly. I hope COLDPower can adapt quickly enough to continue its excellent performance.

Best of luck to you and to all the agent teams in the 2017 competition. I guess one lesson from this episode is that teams should be careful about holding back their best strategies during trial and qualifying rounds, in case those strategies are really exploiting flaws. We are always ready to help teams privately if they suspect their good performance is exploiting a flaw in the simulation. Of course, it's always fair to exploit flaws in your competitors...

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

Ansel
hi

It is possible run several 3 brokers games? 8 brokers sce
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

Ansel
Sorry,


It is possible to run several 3 brokers games? 8 broker games and 3 brokers games are different contexts for our learning algorithms.

thanks in advance

Ansel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

grampajohn
Administrator
Ansel wrote
It is possible to run several 3 brokers games? 8 broker games and 3 brokers games are different contexts for our learning algorithms.
That would require Govert to set things up. He is in Rotterdam, so I'm sure he has gone home for the day. I lack the knowledge and credentials to do it myself.

Note that the prohibition against modifying your broker during the final round does not preclude your brokers from keeping (and sharing) a database that they can use to do autonomous cross-game learning.

Best of luck -

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

susobhang70
In reply to this post by gbuijs
Hello,

I would like to know two things:

#1. If I got the gist correctly, replay of the 26 games will happen. But as per the schedule, it started from game 27. Replays will definitely happen I hope?

#2. I'd like to see the 8 player game trial results which happened yesterday (the end profit scores). I can't seem to be able to find that page. I don't want to download all the logs and parse them to see those results, hope you understand.

Thank You
Susobhan
VidyutVanika Team
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

gbuijs
Hi Susobhan,

Re 1: there hasn't been a decision about replaying : Vedran and Wolf should decide that.

Re 2 : you can download CSV of the round and the games here :
http://ts.powertac.org/log/test/games.4.csv
http://ts.powertac.org/log/test/round.4.csv

Cheers,

Govert
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

susobhang70
Hello govert,

Thank you for the swift update. However, these csv files don't exactly specify what were the broker balances per game. I would like to see the final balances per game like we can for the current round (like here - http://ts.powertac.org:8080/TournamentScheduler/faces/round.xhtml?roundId=4).
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

gbuijs
Hi,

The final balances are in the games CSV.
It doesn't show the broker names, instead it uses broker ids.
The ids can be linked to the names via the round CSV.

The CSV doesn't normalize the result.
But all the games are the same type anyway.

Cheers,

Govert
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

Ansel
Hi all,

It is a fact that given the current configuration of the games, the best or a good strategy for games of 8 brokers seems to be do nothing. In general all brokers loses money. However it is legal no play the games or play doing nothing?

The current configuration stimulates do nothing and rewards disconnections (for games of 8 brokers).

Ansel
COLDTEAM
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Finals 2017 - configuration problem

serkan
@Ansel,

In principal, if the business model is not profitable for any possible parameter set, then better to quit from that business model, which means not to publish any tariff. However, we never know before trying these sets to see whether they are profitable or not. And each trials have typical costs, like publication cost etc. We may loose money during these trials, but we can carry the learned experience to other games.

@Govert

Could you delete the points from the cancelled games? We want to see the situation clearly.

Best,
Serkan
Loading...