Need clear explanation of market-related data types

classic Classic list List threaded Threaded
40 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Need clear explanation of market-related data types

grampajohn
Administrator

There are a number of data elements in powertac-common, and the need for many of them is quite mysterious to me. Could someone please write up a clear explanation of how the market is supposed to work, and include the meanings and purposes of:

  • Shout
  • Orderbook
  • PositionUpdate
  • Product
  • CashUpdate
  • Timeslot
  • TransactionLog
  • CashDoUpdateCmd
  • PositionDoUpdateCmd
  • ShoutDo[Create,Update,Delete]Cmd
That is 12 types. It seems like extreme overkill to me. I think I would reduce this set in half to Bid (or Shout), Timeslot, Orderbook, Position, MarketTransaction, and AccountStatus. I'm actually not sure about Timeslot. All of these would be immutable, so there would be no need for the Cmd types.

What am I missing? I honestly do not know what most of these are for. When I look at the code I see long sequences of code that's copying fields from one structure to another. This makes me suspect redundancy.

Thanks.

John

Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

chris.flath
I think most of the confusion stems from explicit modeling of transitions and using commands instead of constructors for creation. An example of how I understand these different objects:

A shout (that is an order but the term is restricted in grails) for a product (that is an amount of energy in a specific timeslot) is created by means of a ShoutDoCmd, placed and enters the orderbook. The execution of the shout creates a PositionUpdates and CashUpdates for the involved brokers which are then capsulated in UpdateDoCmd and written into the transactionLog.

I don't want to interfere with Daniel's work so far so I cannot decide which objects should be kept or not.

On a quick inspection your short list seems to make a lot of sense John.
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
On 03/09/2011 03:15 AM, chris.flath [via Power TAC Developers] wrote:

> I think most of the confusion stems from explicit modeling of
> transitions and using commands instead of constructors for creation. An
> example of how I understand these different objects:
>
> A shout (that is an order but the term is restricted in grails) for a
> product (that is an amount of energy in a specific timeslot) is created
> by means of a ShoutDoCmd, placed and enters the orderbook. The execution
> of the shout creates a PositionUpdates and CashUpdates for the involved
> brokers which are then capsulated in UpdateDoCmd and written into the
> transactionLog.

It's specifically the cash updates, and possibly the position updates,
that I think are out-of-scope for the market. These types and the
related market functions were developed before the accounting function
was separated from the market. I think Shout, Orderbook, and
MarketTransaction are appropriate for the current scope. I'm not sure
about any of the others, especially if we decide that the current market
position of each agent is maintained by Accounting (I'm beginning to
think that makes sense). Also, I don't think the result will be Updates
- they can be replaced by an hourly account summary, accompanied by the
transactions from the three markets (forward market, balancing market,
and tariff market) that produce the changes from the last accounting.

Thoughts?

John
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

Daniel Schnurr
In reply to this post by grampajohn
After the deletion of the ShoutDoCreate/Delete/Update commands/notifications how is the processShout() method in the ForwardMarket expected to work? Is there an explanation somewhere? I could not find anything on the wiki.

How is the action specified that the market has to handle? And does the broker submit a complete Shout object?

Thanks for your support.
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
Daniel Schnurr wrote
After the deletion of the ShoutDoCreate/Delete/Update commands/notifications how is the processShout() method in the ForwardMarket expected to work? Is there an explanation somewhere? I could not find anything on the wiki.

How is the action specified that the market has to handle? And does the broker submit a complete Shout object?
That is a good question. I never understood what the Delete and Update types were for. The fields that the broker would not use are the execution fields. The dateMod and modReasonCode fields don't apply any more, do they?

Here's what I thought would happen: Broker submits Shout. The executionPrice and executionQuantity fields are not used, can go away, along with dateMod and modReasonCode. When the market makes a match, it creates a MarketTransaction (by calling the Accounting API), and changes the Shout quantity. Once the quantity goes to zero, the Shout is no longer active.

What am I missing?

John
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

Daniel Schnurr
The intention of ShoutDelete and ShoutUpdate was to enable brokers to modify their shouts as long as there is no matching or the timeslot gets outdated. In the case of a periodic market clearing you may create a shout for timeslot t in timeslot t-5. If you notice in t-3 that you don't need the shout anymore you should be able to delete the shout that you made in the past. Same logic does apply to modifications of the quantity (=ShoutUpdate).

We thought about two ways to implement it:
i) Brokers submit Shout objects with an action flag that states "Add/Delete/Update"
ii) Brokers submit Shout commands like "ShoutDoCreateCmd/ShoutDoUpdateCmd/ShoutDoDeleteCmd". Since this approach has the advantage that it only requires the broker to enter the information needed e.g. in the case of a deletion just the shout id and not the whole Shout object, we decided to implement the notification/command objects.

How do we want to do this without having the different cmds nor an action attribute?

If the audit logging is working properly I guess the dateMod can be removed. ExecutionQuantity was created in order to display partially executed orders directly in the Shout object. You may be able to track this also indirectly by analyzing Shout ids and MarketTransactions. The executionPrice is there to record if there is a difference between the original limit order price and the actual execution price (what will be the standard case in a pda).

The purpose of the modReasonCode property is to record the actions performed by the market system. By this means it was easy to track the state of a Shout object at a given time. This was useful within the running simulation (that may be now partly taken over by the audit logging) but also in the post game analysis.
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
In the FERC market structure, the day-ahead market clears once/hour. It's a big optimization problem that works out location-marginal pricing for each PCC. If I'm not mistaken, bids are considered for a single clearing and then discarded. If you want to carry over a bid to the next clearing, you have to resubmit it. So a single bid results in a single (potentially partial) transaction, or it does not. No modifications, no deletions. This is very simple. Is it inadequate?

John
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

chris.flath
This is an adequate implementation for a "oneshot" day ahead market like ferc. The prototype and all of our implementations so far have modelled a forward market like eex where always multiple products ie time slots are traded. It is a simple market which just matches supply and demand not looking at any capacity restrictions and hence on optimization is run upon clearing. Given the distribution grid scope or power tac it always seemed clear to me that this is the type of wholesale market we want to use - as a matter of our European location this is what we know best and would be most interested researching.

Cleary for this market brokers need to be able to revoke and modify open orders when their market assessment changes.

"grampajohn [via Power TAC Developers]" <[hidden email]> wrote:



In the FERC market structure, the day-ahead market clears once/hour. It's a big optimization problem that works out location-marginal pricing for each PCC. If I'm not mistaken, bids are considered for a single clearing and then discarded. If you want to carry over a bid to the next clearing, you have to resubmit it. So a single bid results in a single (potentially partial) transaction, or it does not. No modifications, no deletions. This is very simple. Is it inadequate?

John


________________________________
If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/Need-clear-explanation-of-market-related-data-types-tp2647977p2696434.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here<
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

Daniel Schnurr
We also should keep in mind that we wanted to be able to integrate also other market mechanisms, especially a continous double auction. I think that for this auction type a change/delete functionality is even more important.

But also if you think of block orders like they are possible in the EEX, you have to be able as a Broker to react and modify your bids for future timeslots after a market clearing because the bidding strategy is likely to be dependent if the block order is executed or not in the current timeslot.
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
It seems infeasible to model a viable CDA market with Power TAC, given a time-compression ratio of 720 and highly variable network latencies. I am not in favor of burdening the simulation with substantial complexity that has little chance of being useful now or in the future. If and when we do figure out how to make a CDA market work, we can add the extra complexity at that time. That's the agile way.

In an FERC market, there can be many timeslots open for bids simultaneously, but all timeslots are cleared periodically, and bids are discarded after clearing.

So as long as we are clearing periodically, the KISS approach would be to accept bids until the market clearing process is invoked, clear the market, run the transactions (and communication results to brokers), and discard the bids. Brokers are of course free to resubmit bids that did not completely clear. If we are concerned that a broker might want a bid to be considered only for a single clearing, say the 10:00 clearing, we could add a field to the bid, but that seems like overkill to me.

Make sense?

John


Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
Let's keep in mind that bids in these markets (both EEX and FERC markets, AFAIK) are much more complex than what we have modeled - they are piecewise-linear or step functions, not single quantities and prices. We have been assuming that we could simulate a piecewise-linear function with multiple bids, and perhaps that's correct. But before we get to modifiable bids, we should first update our models to the richer representations used by these markets.

So we have two cans of worms that we might need to deal with eventually. Let's leave them both aside until we have a complete server running.

If we need to do modifiable bids eventually, we can do it the way modifiable tariffs are now being done. See TariffMarketService for details - it only allows two attributes to be changed, with separate operations. This is much simpler than resubmitting tariffs and having to worry about all possible modifications.

John
Reply | Threaded
Open this post in threaded view
|

Re: Need clear explanation of market-related data types

grampajohn
Administrator
In reply to this post by grampajohn
If we are clearing the market and discarding bids in each timeslot, then what, exactly, is an orderbook? Is it just a report of unmatched bids from the most recent clearing? Is it a list of both cleared and uncleared bids? Do we need it?

Just wondering...

John
Reply | Threaded
Open this post in threaded view
|

RE: Need clear explanation of market-related data types

Wolf
Administrator
In reply to this post by grampajohn

 

 

From: grampajohn [via Power TAC Developers] [mailto:[hidden email]]
Sent: Friday, March 18, 2011 03:59 PM
To: Wolf Ketter
Subject: Re: Need clear explanation of market-related data types

 

Let's keep in mind that bids in these markets (both EEX and FERC markets, AFAIK) are much more complex than what we have modeled - they are piecewise-linear or step functions, not single quantities and prices. We have been assuming that we could simulate a piecewise-linear function with multiple bids, and perhaps that's correct. But before we get to modifiable bids, we should first update our models to the richer representations used by these markets.

I think this makes a lot of sense. We should work first on the richer representation, and then we can make it arbitrarily detailed as we want. I’m actually interested to research both markets, i.e. FERC and EEX, and probably others in the future.

So we have two cans of worms that we might need to deal with eventually. Let's leave them both aside until we have a complete server running.

Agreed, especially in the light of our upcoming deadline!

If we need to do modifiable bids eventually, we can do it the way modifiable tariffs are now being done. See TariffMarketService for details - it only allows two attributes to be changed, with separate operations. This is much simpler than resubmitting tariffs and having to worry about all possible modifications.

OK.

Thanks,

Wolf



John


To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.



Disclaimer
De informatie verzonden in dit e-mail bericht inclusief de bijlage(n) is vertrouwelijk en is uitsluitend bestemd voor de geadresseerde van dit bericht. Lees verder: http://www.eur.nl/email-disclaimer.

The information in this e-mail message is confidential and may be legally privileged. Read more: http://www.eur.nl/english/email-disclaimer.

Reply | Threaded
Open this post in threaded view
|

RE: Need clear explanation of market-related data types

Wolf
Administrator
In reply to this post by grampajohn

Usually an orderbook is created every day (hour, 15min, …) anew and is based only on your customers, i.e. the acquired loads and consumption needed. Unmatched bids are bundled in form of a report and may need to get included in the new orderbook.

 

Cheers,

 

Wolf

 

From: grampajohn [via Power TAC Developers] [mailto:[hidden email]]
Sent: Sunday, March 20, 2011 08:54 PM
To: Wolf Ketter
Subject: Re: Need clear explanation of market-related data types

 

If we are clearing the market and discarding bids in each timeslot, then what, exactly, is an orderbook? Is it just a report of unmatched bids from the most recent clearing? Is it a list of both cleared and uncleared bids? Do we need it?

Just wondering...

John


To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.



Disclaimer
De informatie verzonden in dit e-mail bericht inclusief de bijlage(n) is vertrouwelijk en is uitsluitend bestemd voor de geadresseerde van dit bericht. Lees verder: http://www.eur.nl/email-disclaimer.

The information in this e-mail message is confidential and may be legally privileged. Read more: http://www.eur.nl/english/email-disclaimer.

Reply | Threaded
Open this post in threaded view
|

AW: Need clear explanation of market-related data types

chris.flath

I think we are mixing up things here – the orderbook Daniel was implementing is completely wholesale market-specific, i.e. no customer loads show up here. For every product (i.e. timeslot) in the wholesale market we typically have an orderbook that collects all open orders for this product. Although we may not need this kind of collection in a single clearing market design I am strongly in favor of implementing it anyhow as it will be required in an EEX-style continuous market design.

 

Chris

 

Von: Wolf [via Power TAC Developers] [mailto:ml-node+[hidden email]]
Gesendet: Montag, 21. März 2011 09:15
An: Christoph Flath
Betreff: RE: Need clear explanation of market-related data types

 

Usually an orderbook is created every day (hour, 15min, …) anew and is based only on your customers, i.e. the acquired loads and consumption needed. Unmatched bids are bundled in form of a report and may need to get included in the new orderbook.

 

Cheers,

 

Wolf

Reply | Threaded
Open this post in threaded view
|

Re: AW: Need clear explanation of market-related data types

grampajohn
Administrator
chris.flath wrote
I think we are mixing up things here - the orderbook Daniel was implementing is completely wholesale market-specific, i.e. no customer loads show up here. For every product (i.e. timeslot) in the wholesale market we typically have an orderbook that collects all open orders for this product. Although we may not need this kind of collection in a single clearing market design I am strongly in favor of implementing it anyhow as it will be required in an EEX-style continuous market design.
First, we can use the term Product interchangeably with Timeslot only after we have completely eliminated the Product type from our model. I think it's time to do that.

Are you then in agreement that the orderbook is a report of unsatisfied (open) orders? Would this include orders from the wholesale side, or just from other brokers?  I think before we can answer that, we need a clearer notion of the behavior of an abstract buyer on the wholesale side of the market.

Do we also want to report the cleared orders? To me this makes more sense, because it would make the market transparent. Once the orders include power-type attributes then that information would also show up in the orderbook, I assume.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: AW: Need clear explanation of market-related data types

chris.flath
grampajohn wrote
First, we can use the term Product interchangeably with Timeslot only after we have completely eliminated the Product type from our model. I think it's time to do that.
I was repeating the term product just to make the exposition easier to read.

Relating this thread to the discussion on green energy wholesale market I could see that we could have one wholesale market with two products (green, conventional) x 24 timeslots. But apart from this you are right that we do not have different products.
Reply | Threaded
Open this post in threaded view
|

RE: Need clear explanation of market-related data types

Clemens van Dinther
In reply to this post by Wolf
from my perspective it is simpler and straight forward to implement a periodically clearing continuous market (which we allready have running at present). This leaves us more possibilities to adapt it in the future.

In order to economically evaluate the market results we allready have the corresponding tools which is also a reason for using this market model.

With respect to our deadline I suppose it is better to go ahead as originally agreed. I don't like to change the underlying model from day to day.

Thanks,

Clemens
Reply | Threaded
Open this post in threaded view
|

RE: Need clear explanation of market-related data types

chris.flath
Clemens van Dinther wrote
In order to economically evaluate the market results we allready have the corresponding tools which is also a reason for using this market model.
This is an important point. We always said that privision of analysis tools is crucial for Power TAC to be successful. Using a standard market design and standard data types for the wholesale market allows direct application of typical statistics package from financial markets.

AFAIK this was also a reason behind some of the data fields in the order objects and the order book to match the Reuters financial standards.
Reply | Threaded
Open this post in threaded view
|

RE: Need clear explanation of market-related data types

grampajohn
Administrator
In reply to this post by Clemens van Dinther
Clemens van Dinther wrote
from my perspective it is simpler and straight forward to implement a periodically clearing continuous market (which we already have running at present). This leaves us more possibilities to adapt it in the future.
I don't understand this. What is a "periodically clearing continuous market" and how would it differ from what we are trying to implement - a market that clears periodically, once/timeslot?

Thanks

John
12