Rerunning a tournamet game using the same seed

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Rerunning a tournamet game using the same seed

Daniel
Hi,

I believe Wolf has mentioned at some point that it should be possible to rerun a tournament game in a controlled environment using the same seed, while adding more brokers to it. Is such an option currently implemented?

Thanks,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel wrote
I believe Wolf has mentioned at some point that it should be possible to rerun a tournament game in a controlled environment using the same seed, while adding more brokers to it. Is such an option currently implemented?
Yes, and no. The sim server does indeed have the ability to re-use the random-number seeds from a state log file, but see issue #648. It should be a matter of 2-3 hours work for me to fix that; if you are ready to try it, please let me know. Note that the server should be able to read an entire state log and extract the random number seeds, but you can speed it up considerably by filtering out the lines containing "common.RandomSeed".

Keep in mind that this will essentially re-play a simulation under the condition that the same numbers of draws happen from each random number stream at the same times as in the original simulation. This typically works well to test alternative versions of one of the agents, but may not work as well if you change the number of agents. There may be ways to improve this situation to some extent, for example by drawing a reasonably large, fixed number of values from each stream at the beginning of each slot, then discarding the unused draws. If you want to set up experiments that depend on this feature, and the current behavior is inadequate, please let us know. We would be happy to work with you if you have a reasonably precise idea of the behavior you would like to see.

One other item of infrastructure that is not entirely complete is a general experiment-management framework. The intent is to use the Tournament Scheduler for this; it's part of the design, but the user interface for setting up and browsing experiments has not been written. We have done this before and have a pretty clear idea how it should work. The TS has a very nice scheme for starting and monitoring processes on multiple machines, using Jenkins. The original MinneTAC experiment manager depended on a fairly extensive collection of cron jobs, shell scripts, and python programs on the backend, with a PHP web-app on the frontend and a database between the two. It was a real hack-job, pretty much hard-coded to its environment, but it worked really well. You could set up an experiment to run over 100 SCM games over 20 or more machines, with different setups, and it would go off and run them and email you when it was done a few days later.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel and all -

Issue #648 is now fixed and tested in the master branch. It is not deployed, so you need to be working in a server source environment, and you will need to make sure you have pulled down the latest changes to the modules listed in the issue writeup. I have also updated the README to explain exactly how to use this feature.

If you try this out and it does not work as you expect, please let me know.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
Hi John,

Thank you very much for adding this fix.

I believe I tried using this feature, and it seems that it didn't work for me. It is possible that I am doing something wrong. Here is what I did:
1) mvn -Pcli -Dexec.args="--sim --jms-url <hostname>:61616 --brokers mybroker --boot-data bootstrap-data"

This generated the 'powertac-sim-0.state' log file.

2) mvn -Pcli -Dexec.args="--sim --jms-url <hostname>:61616 --brokers mybroker --boot-data bootstrap-data --random-seeds log/powertac-sim-0.state --log-suffix replay1"

This generated the 'powertac-replay1.state' log file

3) mvn -Pcli -Dexec.args="--sim --jms-url <hostname>:61616 --brokers mybroker --boot-data bootstrap-data --random-seeds log/powertac-sim-0.state --log-suffix replay2"

This generated the 'powertac-replay2.state' log file

4) I compared data from 'powertac-replay1.state' and  'powertac-replay2.state' and it seems that there are differences. For instance, in the weather forecasts. Also, it seems that customer consumption is sometimes similar, but customer split between tariffs is not the same.


I have put the server log files in http://www.cs.utexas.edu/~urieli/forJohn/

Please let me know if any other information is needed.

Thanks,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel -

Thanks for your careful work. Unfortunately, the current temporary hack we are using in lieu of real weather forecasts is messing you up.

The weather forecasts are not repeated, because they are currently artificial and do not use seeds from the sim server. The best fix for this will be to use real forecasts; this is issue #645. An alternative would be to include a seed value in the query. Govert might have an opinion about that; when he told me he was generating artificial forecasts, the connection with repeatability did not occur to me.

I do not know whether weather forecasts will affect customer subscription behavior or not, but it would not surprise me, since at least for some customer models the evaluation involves creating a load profile (which can involve weather data) and testing it against different tariffs. Perhaps Prashant or Antonios could respond?

I also do not know whether customer consumption is affected by weather forecasts, but it seems unlikely. It can be affected by tariff details, however, so once subscriptions diverge there could be differences caused by that. Again, Prashant and Antonios might have more insight.

We fully intend this feature to work as advertised, and clearly it does not. If there's a small change we can make to decouple customer subscription evaluation from weather forecasts, it may be worth doing. We will do some checking and let you all know.

Sorry for the confusion -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
Hi John,

OK - thanks for your detailed response.

I will look forward to any updates on it.

I might have a small suggestion: perhaps I misunderstood, but wouldn't it be easier to record the seed used for weather forecasts, rather then decouple the subscription evaluation from weather forecasts? (Especially since subscrition evaluation based on weather forecasts sounds like a necessary, realistic, feature for the simulation)

Best,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

gbuijs
Hi,

- actual weather forecasts : i've been trying to get my hands on actual forecasts, but it seems the weather services aren't too keen on sharing those. I'm not hopeful we'll get actual forecasts. A possible solution is creating and recording the forecasts ourselves. We should take another look at the algorithm Erik created to see if it suffices our needs.

- replaying with recorded weather data : it's possible to run boots and sims with a recorded weather file via the --weather-data option. You can either create an XML file to be used by both the boot and sim, or re-use the state log of the first sim in the subsequent re-runs. More info in the server-distribution readme.

Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Prashant Reddy
In reply to this post by grampajohn
At least in the factored-customer models, capacities DO depend on weather.  Most of the consumption models are affected by temperature.  The wind and solar models are affected by wind speed/direction and temperature/cloud-cover.

Cheers,
Prashant

Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Prashant Reddy wrote
At least in the factored-customer models, capacities DO depend on weather.
 Most of the consumption models are affected by temperature.  The wind and
solar models are affected by wind speed/direction and
temperature/cloud-cover.
Thanks, Prashant. But the question is whether tariff subscription decisions might be influenced by weather forecasts. If not, then there might be a place where you (or Antonios) is using random numbers that are not taken from controlled seeds.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Prashant Reddy
Oh right, and the answer for that is also yes.  As you hinted earlier, subscription decisions are indeed based on capacity forecasts, which in turn depend on weather forecasts.


Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
John, Prashant, Govert,

Thanks for your responses.
That might clarifies why customer subscriptions were not the same between the two runs.

Govert,
Since I am simulating the same dates, weather data is the same - only the weather forecasts differ.

Thanks again,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel wrote
Since I am simulating the same dates, weather data is the same - only the weather forecasts differ.
So it sounds like the answer, at least in the short term, is to get the weather data from a file. Govert has explained how to do that, so perhaps you can try it and let us know how well it works. I'm sure Govert will be willing to help out if you have trouble composing or reading the file, but my understanding is that it just takes data from an old state log, just as we do for the random number seeds.

Please keep us posted on this if you decide to try it.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
OK - I got it.
I tried it and it seemed that weather + weatherForecasts are successfully reproduced in the replay run.

The next discrepancy I found was in the market mWh asks (prices are OK), and some bids.
I prepared a small case that demonstrates the diff between the original run and the replay run:
www.cs.utexas.edu/~urieli/forJohn/market.orig
www.cs.utexas.edu/~urieli/forJohn/market.replay

Thanks,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel and all -

It is good to chase this to ground. I will not have time to look at this in detail until Saturday at the earliest, but in the meantime consider that the simulation is likely not the only source of randomness in the system - there's also the brokers. For a careful analysis of a similar setup, see Eric Sodomka's 2007 AAAI paper.

I'll put this on my stack and try to get to it latest early next week.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
John,

OK - thanks for your response, and thanks for sending the paper.

Just to clarify: I am currently running a simulation with only one broker (in addition to the default broker).

Thanks again,
Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel wrote
Just to clarify: I am currently running a simulation with only one broker (in addition to the default broker).
Does that one broker also repeat its random sequences? I assume you thought of that, right?

Just checking -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
Hi John,

In fact, I am using the sample broker, and didn't remember it had a random element. However, since it had most of the customers, and since the total consumption is usually much higher then the total production, it never submitted asks - so the discrepancy in the asks values possibly came from somewhere else. Actually, could randomized broker bids have caused the discrepancy in Genco's asks between the two runs?

Thanks again,
Daniel

Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel wrote
Could randomized broker bids have caused the discrepancy in Genco's asks between the two runs?
Yes, of course. Each genco, like the broker, tries to sell its remaining capacity in each of the 24 clearings for a given timeslot. So it is possible that in game 1 a genco sells some timeslot-10 capacity to the broker in timeslot 6, but in game 2 the broker's bids are a bit higher and the genco sells that same capacity in timeslot 5. After that, all the bids and asks for that timeslot will likely be different.

Does this make sense?

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

grampajohn
Administrator
Daniel and all -

I think we need to be clear about what you can expect from the technique of repeating random sequences in a simulation like Power TAC. That is precisely what the Sodomka paper is about. In the context of TAC SCM, he showed that the statistical noise could be significantly reduced, but not eliminated, for precisely the same reason you are seeing with Power TAC. That is a result of randomness outside the scope of your control.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Rerunning a tournamet game using the same seed

Daniel
In reply to this post by grampajohn
Hi John,

Thanks - it makes sense.

Just FYI, I reran another simulation, using one broker, with deterministic bids, and it seems that GenCo's mWh ask amounts differ right from their first order - before the broker submits any bids.

Thanks again,
Daniel
12