New schedule for qualifying round

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

New schedule for qualifying round

grampajohn
Administrator
Dear colleagues -

It appears that the tournament scheduler and the broker login process still need some work and coordination. Wolf suggests that we re-schedule the qualifying round to start on Tuesday 5/29 and run through Friday 6/1. That will give us time to do some cleanup and for broker developers to more thoroughly test their brokers against the tournament setup. More importantly, it will let us do this without being in continuous panic mode, which is taking a toll. If this is terribly inconvenient for any of you, please let us know as soon as possible. Otherwise, you should plan on starting next Tuesday.

Please don't lose hope. It appears that the simulation server itself is quite solid, as long as the brokers are reasonably well-behaved. We are not seeing errors or exceptions once they get started. It also appears that the basic tournament deployment setup is quite robust. The problems seem to be focused in the scheduling function and in server management. Due to time pressures, and to the complexity of what we are trying to do, some of the basic maintenance functions are not yet surfaced in the admin web interface - you have to dig into the database and run manual queries, and right now Erik is the only person who knows it well enough to do that.

There is also a potential problem with the visualizers getting out of sync with their respective servers when we manually kill the servers, usually due to broker problems. I'm looking into implementing a periodic "probe" from the visualizer so it can tell whether a server is still alive when no message traffic is arriving. Unfortunately, it seems that we may have broken several of the visualizers last night while we were manipulating server processes.

We will keep posting individual short games (any game with "test" in its name is a short game), and a few short multi-game tournaments over the next few days, to give you as many opportunities as possible to test your agents.

If you see something that does not seem right, please let us know. We will get this working; it's a small piece of a large ensemble, most of which seems quite solid.

Again, thank you for your patience. We are very focused on having a successful and trouble-free tournament, even though it's taking longer to get there than we had hoped.

John and Erik
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Hi John,

This is a good news. Thanks.
Can you please provide us the new or the final version of the Power TAC server (probably the 0.5.2).

So, we will be able to locally test the new server.

Many thanks,

Rod
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
Rodrig wrote
Hi John,

This is a good news. Thanks.
Can you please provide us the new or the final version of the Power TAC server (probably the 0.5.2).

So, we will be able to locally test the new server.
So far, there are few broker-visible changes in the server - just some adjustments in customer subscription behavior - but I'll look into doing another release. It takes several hours to do a release, I'm travelling tomorrow and Sunday, and I am not yet prepared for the trip. So it might not happen until after I arrive in Rotterdam.

In the meantime, you are welcome to run 0.6.0-SNAPSHOT. If you want to try that, let me know and I'll post a distribution package. It's all deployed, because that's what the tournament scheduler is using.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Hi John,

I am very sorry for the inconvenience.

I am already using the PT version 0.5.1 or the 0.6.0-SNAPSHOT. During my Test today against the tournement server, one of my agents had an error and I am not able to debug it.

And I saw that there is a new server component "windfarm genco" that is not available in the current PT version. I actually suppose it may be interesting to test the agent with the new version of the server.

The alternative may be to provide us more test opportunities. I think that would be enough.

Many  thanks,

Rod
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
Rodrig wrote
Hi John,

I am very sorry for the inconvenience.
No problem. We need to hear from you to make sure we are spending our time where it will do you the most good.
I am already using the PT version 0.5.1 or the 0.6.0-SNAPSHOT. During my Test today against the tournement server, one of my agents had an error and I am not able to debug it.
Yes, the most common errors we are seeing are logging in to the wrong game, failing to respond correctly to the BrokerAccept message that follows a successful login, or the agent not receiving messages, which causes the server's output queue to fill up and wedge the server. We think we have a solution to that last problem, and will be implementing it in the next day or so.

One thing that's not implemented yet is giving you convenient access to the game logs. I'll have to look into what's involved with that. Unfortunately, if a server does not finish and exit correctly, the game logs are not collected and posted, although in most cases those of us with direct access to the machines can find them.
And I saw that there is a new server component "windfarm genco" that is not available in the current PT version. I actually suppose it may be interesting to test the agent with the new version of the server.
The windfarm genco is under development, primarily by Shashank Pande and Ryan Finneman. We hope to have it ready for the finals, but unfortunately it's still incomplete, and I have had no time to help out with it. We think it will be an interesting addition, because of it's bidding behavior based on weather forecasts. There are a couple of papers on that; I'm sure Shashank could point you to them if you are interested.
The alternative may be to provide us more test opportunities. I think that would be enough.
That we will do, and we expect it will be more stable and usable every day. When we are actively developing, we are updating scheduler frequently, and it's a pain to do that when there are games running.

Thanks for your input.

John
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Thank you very much for the detailed answers,

Rod
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
In reply to this post by grampajohn
Dear colleagues -

We have made considerable progress, and it will soon be time to test the multi-game scheduling function. The plan is to set up a small multi-game tournament sometime tomorrow, and shoot for a full multi-day tournament on Wednesday. The timing tomorrow will depend on how confident Erik is that it's ready. If he's confident this evening before he has to quit for the day, he will set it up and make an announcement. Otherwise we will try to set it up after he has started his day tomorrow and he and I have had a chance to look at it. That could be sometime after 14:00 UTC. For a test tournament, it is not essential that everyone participate, but it will be important to have multiple brokers, preferably at least six.

More news tomorrow, if not later tonight. I'm still suffering a bit from jet lag and need to get to bed.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

markus
 > The plan is to set up a small multi-game tournament sometime
tomorrow, and shoot for a full multi-day tournament on Wednesday.

Just to be sure: the idea here is to have multiple instances of each
broker ready, each of which plays at most one game upon request of the
tournament scheduler, correct? Put another way: there is no need for the
broker to recycle itself after the end of a game.

Cheers,

Markus
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
markus wrote
Just to be sure: the idea here is to have multiple instances of each
broker ready, each of which plays at most one game upon request of the
tournament scheduler, correct? Put another way: there is no need for the
broker to recycle itself after the end of a game.
Very good question, Markus -

The idea is that each team would have two copies of their broker available, with the same name and credentials. Both would be started at the beginning of a multi-game competition, and would repeatedly attempt to log in until the scheduler creates their games. Once a given agent completes a game, it should again start attempting to log in for its next game. You can do that by recycling the agent, or by writing a script that starts it up again after it exits at the end of a game. The script I have used for SCM also compresses and moves the logfiles to a different directory before restarting the agent. If anyone is interested, I could probably dig up a copy of that script.

The scheduler has a goal of running multiple sets of games of different sizes, in such a way that each agent plays the same numbers of games of each size, constrained by the available servers and by the number of copies of each agent that are available. Because the game length is stochastic, these cannot be scheduled ahead of time; instead, we have an incremental algorithm that chooses a game from the set of unfinished games whenever resources become available. It tries to be fairly smart about it, so it does not end up with all the 8-player games at the end, at which point they would have to be run serially if there are fewer than 16 different brokers. Once a game is chosen, it is assigned a boot record and started, and the associated brokers are given the credentials to log into it.

Does this answer your question?

John
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Hi John,

Your script is very wellcome,

Many thanks,

Rod

Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

icmoon
Dear John:

First of all, thanks for the effort to make this competition successful.

It will be great help for us to have your script for iterative runs.

Thanks,

Il-Chul Moon

Assistant Professor
Dept. of Industrial and Systems Engineering
KAIST


On Tue, May 29, 2012 at 8:30 PM, Rodrig [via Power TAC Developers] <[hidden email]> wrote:
Hi John,

Your script is very wellcome,

Many thanks,

Rod




If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/New-schedule-for-qualifying-round-tp4015421p4020531.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
In reply to this post by Rodrig
On 05/29/2012 06:30 AM, Rodrig [via Power TAC Developers] wrote:
> Hi John,
>
> Your script is very wellcome,

I have posted it at
https://github.com/downloads/powertac/powertac-server/runMany.sh. The
actual running happens in ./run, which starts the agent, waits for it to
finish, then compresses and moves the logs. I could post that also, but
it uses ant and is not very informative.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Sorry, but how to run my agent automatically many times if I am using STS.

Thanks a lot,

Rod

 
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
On 05/29/2012 08:39 AM, Rodrig [via Power TAC Developers] wrote:
> Sorry, but how to run my agent automatically many times if I am using STS.

STS is a development environment. If you have an agent that works, you
should be able to run it outside STS, using for example maven or ant.
The sample broker shows how to do that using maven.

An alternative is to run it inside STS, and arrange to re-initialize it
every time a game ends, so it closes the old log files, opens new ones,
and starts the login process again.

Is this what you were looking for?

John

Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

Rodrig
Thanks a lot John,

The problem is that I am using Windows and I am not quiet confortable with the batch file command. But I am experimenting it.

Thanks,
 

Rod
Reply | Threaded
Open this post in threaded view
|

Re: New schedule for qualifying round

grampajohn
Administrator
In reply to this post by markus
Colleagues -

If you have been watching the tournament site, you will have seen some multi-game tournaments in the schedule. Erik is testing the final links in the chain to actually running games under the incremental scheduler, and it's really hard to do that on a single machine. He is in timezone UTC-5, and he will set up some test games before he signs off that you are welcome to use to try out your brokers. If all goes well, we'll put up a short multi-game tournament tomorrow and ask as many of you as possible to sign up. We'll then analyze the behavior and take your input, and hope to start a longer tournament, giving everyone enough time to sign up, by late tomorrow or early on Thursday.

Erik intends to send out a message before he signs off to let you know that the scheduler is ready for test usage.

As always, your comments and questions are important to us.

John