Here are two suggestions/questions that we have for Powertac 2018. Please consider if they are feasible.
a) The Powertac finals roughly spans over a week or ten days (or more). Occasionally, there could be breaks due to maintenance. Our understanding is that a particular team is not expected to change the broker between games during finals. I find this rule is difficult to enforce by the organizers. It is possible that teams can change broker between games without losing connection to Powertac server. In this context, please consider the following way-outs:
i) Relax the rule. Is it possible that the organizers could relax this rule but keeping the condition that an instance (or two) of the broker should always be available whenever a game is scheduled ?
ii) Enforce the rule. Enforce the rules by requesting teams to give their Jar file and organizers can run them on their behalf (this is too much work for the organizers)
I am not sure how else can the organizer enforce the rule. The reason for our suggestion is that, if there is a rule, it should be enforceable; otherwise, teams that follow the rule can be at disadvantage.
b) It is not very clear what is the consequence if a broker gets disconnected in a game or two and as a consequence has zero profit for those games. There may be other brokers who participated and got negative profits (or loss). Would such loss making brokers be at a disadvantage compared to brokers who simply did not play the game ? Does the z-score computation takes care of these eventualities ? If not can we remedy that ?
Extending the above line the thinking, can brokers as part of their strategy, be connected to the simulation server, but don't participate in any transactions (like tariff, wholesale or balancing) to keep their balance sheets clean, in some of their games ?
The "no modification during a final round" rule is indeed hard to enforce, and if there were cash prizes for winning we should expect some level of cheating. But in fact we have enforced it at least once that I know of. Possibly Wolf or Govert know of others. In the case I know about, the behavior of one of the agents seemed to change noticeably partway through a tournament. One of the other participants, who was paying closer attention to agent performance than we were, called our attention to it. It did appear that there was a definite change, not a gradual improvement through the tournament as one might expect from a learning agent. So we talked to the team, and were told that one of the machines running their agent copies failed, and they started up an agent on another machine. But they had been working on the agent, and so the one they started was a newer version than the one that had been running. Quite possibly the student who started the agent was not aware that the code had been updated, but the team volunteered to be removed from the rankings to avoid hard feelings among the other teams.
My feeling is that we are a community of researchers, working toward a common goal. Of course we all have our own goals as well, and it's always nice to be able to write a paper with "Champion Agent" in the title. At the same time, we all care about our reputations among our peers. So we cannot stop you from cheating, but if you do and you are successful, others in the community will at least suspect that you cheated, and may very well post the details of what they have observed where the rest of us can see it. I think that's a pretty strong disincentive.
I'll let Wolf and/or Govert address the issue of disconnection. However, I will say that sometimes the best strategy is to not participate, if some other agent is making it impossible to be profitable. In such a situation, a well-designed agent might decide to sit the game out and get a score as close to zero as it can. I see nothing wrong with that.
"Sometimes the best strategy is to not participate, if some other agent is making it impossible to be profitable. In such a situation, a well-designed agent might decide to sit the game out and get a score as close to zero as it can"
But the well-designed agent must stay connected.
I consider that if a connected agent, decides not to send orders in any market to get a score as close to zero as it can (using its intelligence), it is right. However, if the agent designer or the agent owner, decides disconnect the agent to get a score as close to zero as it can, it is wrong.
I consider that repeated disconnection should be penalized in the overall score.
It is understandable that any agent may have a network problem that implies a temporal disconnection. But disconnection as a strategy is not a agent strategy, it is a strategy the owner of the agent. I consider that on final round, the agent must make your own decisions, with out any action of the the owner of the agent.
On the other hand, I have very personal opinion. I prefer a simulation configuration that encourages the agent learn to do something that a simulation configuration that encourages the agent learn to do nothing. But it is debatable and interesting.
I agree that intentional disconnecting should be an invalid strategy.
But it would be hard to enforce, identical to "no modification" rule.
Brokers might experience unhandled exceptions and exit.
Some participants have unreliable network connections.