Source, form, and granularity of consumption/production records and predictions

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Source, form, and granularity of consumption/production records and predictions

grampajohn
Administrator
I've been looking through the server documentation, and I cannot find anything about how brokers get information about expected power production and consumption. We also have to think carefully about the form and granularity of such information.

One of the original ideas about the MIS was that it would, for a fee, break down the customer population by "preference clusters" and be able to tell brokers about various features of their behavior, such as consumption profiles and price elasticity. Brokers could then build tariffs that somehow target those customer groups in order to maximize the value of their portfolios.

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase. Here's a proposal for a minimal first-cut:

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption.

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data.

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs.

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions.

Is this enough?

John
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Source, form, and granularity of consumption/production records and predictions

grampajohn
Administrator
I propose we call the module that delivers production/consumption records the "Market Data Service".
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Source, form, and granularity of consumption/production records and predictions

Carsten Block
Administrator
In reply to this post by grampajohn
Hi John,

thanks for pointing this out. I like the ideas. Here is what we thought in Karlsruhe. It seems to be very well compatible:

1.)  Upon competition start each customer model publishes its master data record, which will be made available to all brokers. This master data record includes a general description, such as "customer type", "average annual consumption / generation capacity" and so on.

2.) Upon each timeslot (or maybe each day) change "aggregated" statics about historic consumption and production of all customers is made available to everybody (and we thought to keep them on a per customer basis, which gives brokers a clearer picture on which customer consumed / produced how much). For the statistics data we think of boxplot information, i.e. mean, median, std, 25% 75% quantile. Unclear is on how much history to aggregate (only one day or maybe better a week or "type days").

3.) After timeslot deactivation each customer sends out his "meter reading record", which is stored on the server in the accounting service, and forwarded to the distribution utility for computing balancing power. Also the respective brokers get "their" meter readings, i.e. those from their customers. 

So how does that match to your suggestions?

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase.
That's right. So what we call "customer master data" should be really a very rough categorization only (brokers should be able to learn that a customer is a producer, for example, and what is technical wind-to-energy ratio would be).


- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption. 
This is a "more detailed" version of our "aggregate statistics" idea. I think, both is well possible to realize technically. To me, the question is how much data we really want to reveal. I can't judge. We replace the "is available any time" with a "will be updated and sent out to brokers every hour (or maybe every day)".

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data. 
We didn't think of overall aggregated consumption and production data but probably that's a nice value add, in particular if we decide to go with the minimal information solution and only send out the "box plot statistics" for each customer.

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs. 
Here we thought, that they should really get the "raw meter readings". I find that a bit more natural as this is the data they get in reality too with smart meters installed. As said before: Accounting Service stores all raw data for reference and Distribution Utility gets aggregated data on a per broker basis so that i can compute imbalances and impose balancing power fees.

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions. 
Just to avoid misunderstanding: Weather forecasts != predictions?

We understand that you have access to historic weather data as well as corresponding historic weather forecast data (= predictions?!), right? Our idea was to send out the historic forecast data to brokers so that they can set up their own forecasting models given that they know the basic wind-to-power and sun-to-power ratios (or even more elaborate "production functions" as you call it) of their customers (they are published in publicly available customer master data). The true weather data is only sent out  for the "now" timeslot to (i) customers and (ii) brokers. Customer models can use this data to compute their actual production (or maybe even consumption if a consumer uses electric heating or cooling and thus cares about outside temperature). Brokers get this data as feedback for their prediction models. 

Artificially generated demand / production forecasts are no longer sent out.

To sum up: We mainly need to decide, what and how much "aggregate historic" data should be publicly available.

Cheers,
Carsten 



Am 06.12.2010 um 22:02 schrieb grampajohn [via Power TAC Developers]:

I've been looking through the server documentation, and I cannot find anything about how brokers get information about expected power production and consumption. We also have to think carefully about the form and granularity of such information.

One of the original ideas about the MIS was that it would, for a fee, break down the customer population by "preference clusters" and be able to tell brokers about various features of their behavior, such as consumption profiles and price elasticity. Brokers could then build tariffs that somehow target those customer groups in order to maximize the value of their portfolios.

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase. Here's a proposal for a minimal first-cut:

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption.

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data.

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs.

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions.

Is this enough?

John



View message @ http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2029952.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

AW: Source, form, and granularity of consumption/production records and predictions

chris.flath

Hi,

 

very nice summary by you two. I think we are very much on the same page in this issue. My observations and remarks:

 

·         For me the MIS always seemed to be somewhat redundant in a competitive environment as the matching should take place by endogenous consumer decision. Similarly, the brokers should not ask what can I expect with this tariff but actually run the risk of offering it. Hence I do like the current agreed path much better and it seems like a more natural way to describe a marketplace.

·         I really like John’s idea of publishing a population aggregate data – this seems like very natural information for a marketplace as published by a federal statistics bureau

·         During our coding week we had some offline discussions on how to generate historic consumption data at time 0. To take this discussion online I wanted to restate the basic solutions we found so far: self-declared consumption data by the designer of the customer class vs. pre-play on the server (eg playing rounds -10 to 0 with only a default tariff offered which provides a flat tariff). The latter approach is favored by me – what do you think?

·         Considering historical consumption, I feel we should not be to detailed in this respect. I think from a game perspective this is more desirable than inducing all brokers to do massive data crawling on an initial dataset. However, I see John’s point that we need to provide the brokers with something so this should be discussed again. It boils down to a matter of taste.

·         I noticed that we should specify (what kind of data, how are forecasts and weather realizations created, what types of weather parameters are modeled) the physical environment engine as soon as possible as it will be important for consumer modeling.

                                                                                                                                                                                   

Cheers,


Chris

 

Von: Carsten Block [via Power TAC Developers] [mailto:[hidden email]]
Gesendet: Dienstag, 7. Dezember 2010 10:36
An: Christoph Flath
Betreff: Re: Source, form, and granularity of consumption/production records and predictions

 

Hi John,

 

thanks for pointing this out. I like the ideas. Here is what we thought in Karlsruhe. It seems to be very well compatible:

 

1.)  Upon competition start each customer model publishes its master data record, which will be made available to all brokers. This master data record includes a general description, such as "customer type", "average annual consumption / generation capacity" and so on.

 

2.) Upon each timeslot (or maybe each day) change "aggregated" statics about historic consumption and production of all customers is made available to everybody (and we thought to keep them on a per customer basis, which gives brokers a clearer picture on which customer consumed / produced how much). For the statistics data we think of boxplot information, i.e. mean, median, std, 25% 75% quantile. Unclear is on how much history to aggregate (only one day or maybe better a week or "type days").

 

3.) After timeslot deactivation each customer sends out his "meter reading record", which is stored on the server in the accounting service, and forwarded to the distribution utility for computing balancing power. Also the respective brokers get "their" meter readings, i.e. those from their customers. 

 

So how does that match to your suggestions?

 

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase.

That's right. So what we call "customer master data" should be really a very rough categorization only (brokers should be able to learn that a customer is a producer, for example, and what is technical wind-to-energy ratio would be).

 

 

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption. 

This is a "more detailed" version of our "aggregate statistics" idea. I think, both is well possible to realize technically. To me, the question is how much data we really want to reveal. I can't judge. We replace the "is available any time" with a "will be updated and sent out to brokers every hour (or maybe every day)".

 

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data. 

We didn't think of overall aggregated consumption and production data but probably that's a nice value add, in particular if we decide to go with the minimal information solution and only send out the "box plot statistics" for each customer.

 

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs. 

Here we thought, that they should really get the "raw meter readings". I find that a bit more natural as this is the data they get in reality too with smart meters installed. As said before: Accounting Service stores all raw data for reference and Distribution Utility gets aggregated data on a per broker basis so that i can compute imbalances and impose balancing power fees.

 

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions. 

Just to avoid misunderstanding: Weather forecasts != predictions?

 

We understand that you have access to historic weather data as well as corresponding historic weather forecast data (= predictions?!), right? Our idea was to send out the historic forecast data to brokers so that they can set up their own forecasting models given that they know the basic wind-to-power and sun-to-power ratios (or even more elaborate "production functions" as you call it) of their customers (they are published in publicly available customer master data). The true weather data is only sent out  for the "now" timeslot to (i) customers and (ii) brokers. Customer models can use this data to compute their actual production (or maybe even consumption if a consumer uses electric heating or cooling and thus cares about outside temperature). Brokers get this data as feedback for their prediction models. 

 

Artificially generated demand / production forecasts are no longer sent out.

 

To sum up: We mainly need to decide, what and how much "aggregate historic" data should be publicly available.

 

Cheers,

Carsten 

 

 

 

Am 06.12.2010 um 22:02 schrieb grampajohn [via Power TAC Developers]:



I've been looking through the server documentation, and I cannot find anything about how brokers get information about expected power production and consumption. We also have to think carefully about the form and granularity of such information.

One of the original ideas about the MIS was that it would, for a fee, break down the customer population by "preference clusters" and be able to tell brokers about various features of their behavior, such as consumption profiles and price elasticity. Brokers could then build tariffs that somehow target those customer groups in order to maximize the value of their portfolios.

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase. Here's a proposal for a minimal first-cut:

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption.

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data.

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs.

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions.

Is this enough?

John


 

 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Source, form, and granularity of consumption/production records and predictions

Prashant Reddy
Hi all,

I've discussed this topic with John offline so I mostly agree with this proposal.  I have three comments/questions:

1) One outstanding question seems to be:  Is the historical data (i) aggregated over time, but available for each customer/customer-model separately or (ii) is it aggregated over all customers, but available for different periods (seasons?) separately, or (iii) is it aggregated over both dimensions?  Maybe there should be a fee for the breakdown by customer while the other two options are free? 

2) Free information published by a federal bureau is typically delayed by a few weeks or few months.  Do we want to offer the option of near real-time information for a fee?  I think the fee is useful if we think that every broker requesting or subscribing to real-time information will overwhelm the server/MarketDataService.

3) Another outstanding question:  How is the initial historical data generated, self-declared or Chris' suggestion of -10-to-0 timeslots?  I liked Chris' idea initially, but thinking some more... if there is only one flat tariff initially, do we need the 10 timeslots?  Won't the customer models be able to deterministically generate their "historical consumption/production" if they simply know what that flat tariff is?  Further, is the preexisting flat tariff known ahead of time to the customer models?  If so, then it seems that the customer models should be able to self-declare their historic data at start-up?

Please let me know if any of the above is difficult to understand.

Thanks,
Prashant


On Tue, Dec 7, 2010 at 8:42 AM, flath [via Power TAC Developers] <[hidden email]> wrote:

Hi,

 

very nice summary by you two. I think we are very much on the same page in this issue. My observations and remarks:

 

·         For me the MIS always seemed to be somewhat redundant in a competitive environment as the matching should take place by endogenous consumer decision. Similarly, the brokers should not ask what can I expect with this tariff but actually run the risk of offering it. Hence I do like the current agreed path much better and it seems like a more natural way to describe a marketplace.

·         I really like John’s idea of publishing a population aggregate data – this seems like very natural information for a marketplace as published by a federal statistics bureau

·         During our coding week we had some offline discussions on how to generate historic consumption data at time 0. To take this discussion online I wanted to restate the basic solutions we found so far: self-declared consumption data by the designer of the customer class vs. pre-play on the server (eg playing rounds -10 to 0 with only a default tariff offered which provides a flat tariff). The latter approach is favored by me – what do you think?

·         Considering historical consumption, I feel we should not be to detailed in this respect. I think from a game perspective this is more desirable than inducing all brokers to do massive data crawling on an initial dataset. However, I see John’s point that we need to provide the brokers with something so this should be discussed again. It boils down to a matter of taste.

·         I noticed that we should specify (what kind of data, how are forecasts and weather realizations created, what types of weather parameters are modeled) the physical environment engine as soon as possible as it will be important for consumer modeling.

                                                                                                                                                                                   

Cheers,


Chris

 

Von: Carsten Block [via Power TAC Developers] [mailto:[hidden email]]
Gesendet: Dienstag, 7. Dezember 2010 10:36
An: Christoph Flath
Betreff: Re: Source, form, and granularity of consumption/production records and predictions

 

Hi John,

 

thanks for pointing this out. I like the ideas. Here is what we thought in Karlsruhe. It seems to be very well compatible:

 

1.)  Upon competition start each customer model publishes its master data record, which will be made available to all brokers. This master data record includes a general description, such as "customer type", "average annual consumption / generation capacity" and so on.

 

2.) Upon each timeslot (or maybe each day) change "aggregated" statics about historic consumption and production of all customers is made available to everybody (and we thought to keep them on a per customer basis, which gives brokers a clearer picture on which customer consumed / produced how much). For the statistics data we think of boxplot information, i.e. mean, median, std, 25% 75% quantile. Unclear is on how much history to aggregate (only one day or maybe better a week or "type days").

 

3.) After timeslot deactivation each customer sends out his "meter reading record", which is stored on the server in the accounting service, and forwarded to the distribution utility for computing balancing power. Also the respective brokers get "their" meter readings, i.e. those from their customers. 

 

So how does that match to your suggestions?

 

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase.

That's right. So what we call "customer master data" should be really a very rough categorization only (brokers should be able to learn that a customer is a producer, for example, and what is technical wind-to-energy ratio would be).

 

 

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption. 

This is a "more detailed" version of our "aggregate statistics" idea. I think, both is well possible to realize technically. To me, the question is how much data we really want to reveal. I can't judge. We replace the "is available any time" with a "will be updated and sent out to brokers every hour (or maybe every day)".

 

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data. 

We didn't think of overall aggregated consumption and production data but probably that's a nice value add, in particular if we decide to go with the minimal information solution and only send out the "box plot statistics" for each customer.

 

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs. 

Here we thought, that they should really get the "raw meter readings". I find that a bit more natural as this is the data they get in reality too with smart meters installed. As said before: Accounting Service stores all raw data for reference and Distribution Utility gets aggregated data on a per broker basis so that i can compute imbalances and impose balancing power fees.

 

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions. 

Just to avoid misunderstanding: Weather forecasts != predictions?

 

We understand that you have access to historic weather data as well as corresponding historic weather forecast data (= predictions?!), right? Our idea was to send out the historic forecast data to brokers so that they can set up their own forecasting models given that they know the basic wind-to-power and sun-to-power ratios (or even more elaborate "production functions" as you call it) of their customers (they are published in publicly available customer master data). The true weather data is only sent out  for the "now" timeslot to (i) customers and (ii) brokers. Customer models can use this data to compute their actual production (or maybe even consumption if a consumer uses electric heating or cooling and thus cares about outside temperature). Brokers get this data as feedback for their prediction models. 

 

Artificially generated demand / production forecasts are no longer sent out.

 

To sum up: We mainly need to decide, what and how much "aggregate historic" data should be publicly available.

 

Cheers,

Carsten 

 

 

 

Am 06.12.2010 um 22:02 schrieb grampajohn [via Power TAC Developers]:



I've been looking through the server documentation, and I cannot find anything about how brokers get information about expected power production and consumption. We also have to think carefully about the form and granularity of such information.

One of the original ideas about the MIS was that it would, for a fee, break down the customer population by "preference clusters" and be able to tell brokers about various features of their behavior, such as consumption profiles and price elasticity. Brokers could then build tariffs that somehow target those customer groups in order to maximize the value of their portfolios.

It's been argued that is it unrealistic to expect such information to be available at the granularity of customers or customer "clusters" defined in any useful way. That's probably true. But that still leaves the question of exactly what information brokers should be able to see or purchase. Here's a proposal for a minimal first-cut:

- Historical consumption and production data is available at any time, including at the beginning of the game, for the entire population. This data is broken down into hourly amounts for weekdays and weekends, for each month of the year. So for a 12-month year, you would get an array 12x2x24 for both production and consumption.

- Actual consumption and production data, aggregated over the entire population, is published at the end of each timeslot. This will allow brokers to build up their own models and compare them with the aggregate historical data.

- Actual consumption and production data is available to individual brokers, at the end of each timeslot, aggregated over each of their existing tariffs.

- No predictions for weather-related production (solar and wind sources) are available. Instead, the output of these sources will be driven by the environment model, with an appropriate level of added noise. Brokers must use weather forecasts along with the published production functions to make their own predictions.

Is this enough?

John


View message @ http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2029952.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

 

 


View message @ http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2032533.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.




View message @ http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2033523.html

To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Source, form, and granularity of consumption/production records and predictions

grampajohn
Administrator
I'm in the process of trying to condense the conversation in the
domain-dta wiki at
https://github.com/powertac/domain-data/wiki/Consumption-and-Production-records-and-predictions.
Feel free to pitch in; I'll have little or no time tomorrow to work on
it until Friday.

John


On 12/07/2010 10:13 AM, ppr [via Power TAC Developers] wrote:

> Hi all,
>
> I've discussed this topic with John offline so I mostly agree with this
> proposal.  I have three comments/questions:
>
> 1) One outstanding question seems to be:  Is the historical data (i)
> aggregated over time, but available for each customer/customer-model
> separately or (ii) is it aggregated over all customers, but available
> for different periods (seasons?) separately, or (iii) is it aggregated
> over both dimensions?  Maybe there should be a fee for the breakdown by
> customer while the other two options are free?
>
> 2) Free information published by a federal bureau is typically delayed
> by a few weeks or few months.  Do we want to offer the option of near
> real-time information for a fee?  I think the fee is useful if we think
> that every broker requesting or subscribing to real-time information
> will overwhelm the server/MarketDataService.
>
> 3) Another outstanding question:  How is the initial historical data
> generated, self-declared or Chris' suggestion of -10-to-0 timeslots?  I
> liked Chris' idea initially, but thinking some more... if there is only
> one flat tariff initially, do we need the 10 timeslots?  Won't the
> customer models be able to deterministically generate their "historical
> consumption/production" if they simply know what that flat tariff is?
> Further, is the preexisting flat tariff known ahead of time to the
> customer models?  If so, then it seems that the customer models should
> be able to self-declare their historic data at start-up?
>
> Please let me know if any of the above is difficult to understand.
>
> Thanks,
> Prashant
>
>
> On Tue, Dec 7, 2010 at 8:42 AM, flath [via Power TAC Developers]
> <[hidden email] </user/SendEmail.jtp?type=node&node=2034337&i=0>> wrote:
>
>     Hi,
>
>     very nice summary by you two. I think we are very much on the same
>     page in this issue. My observations and remarks:
>
>     · For me the MIS always seemed to be somewhat redundant in a
>     competitive environment as the matching should take place by
>     endogenous consumer decision. Similarly, the brokers should not ask
>     what can I expect with this tariff but actually run the risk of
>     offering it. Hence I do like the current agreed path much better and
>     it seems like a more natural way to describe a marketplace.
>
>     · I really like John’s idea of publishing a population aggregate
>     data – this seems like very natural information for a marketplace as
>     published by a federal statistics bureau
>
>     · During our coding week we had some offline discussions on how to
>     generate historic consumption data at time 0. To take this
>     discussion online I wanted to restate the basic solutions we found
>     so far: self-declared consumption data by the designer of the
>     customer class vs. pre-play on the server (eg playing rounds -10 to
>     0 with only a default tariff offered which provides a flat tariff).
>     The latter approach is favored by me – what do you think?
>
>     · Considering historical consumption, I feel we should not be to
>     detailed in this respect. I think from a game perspective this is
>     more desirable than inducing all brokers to do massive data crawling
>     on an initial dataset. However, I see John’s point that we need to
>     provide the brokers with something so this should be discussed
>     again. It boils down to a matter of taste.
>
>     · I noticed that we should specify (what kind of data, how are
>     forecasts and weather realizations created, what types of weather
>     parameters are modeled) the physical environment engine as soon as
>     possible as it will be important for consumer modeling.
>
>     Cheers,
>
>
>     Chris
>
>     *Von:* Carsten Block [via Power TAC Developers] [mailto:[hidden
>     email] <http://user/SendEmail.jtp?type=node&node=2033523&i=0>]
>     *Gesendet:* Dienstag, 7. Dezember 2010 10:36
>     *An:* Christoph Flath
>     *Betreff:* Re: Source, form, and granularity of
>     consumption/production records and predictions
>
>     Hi John,
>
>     thanks for pointing this out. I like the ideas. Here is what we
>     thought in Karlsruhe. It seems to be very well compatible:
>
>     1.)  Upon competition start each customer model publishes its master
>     data record, which will be made available to all brokers. This
>     master data record includes a general description, such as "customer
>     type", "average annual consumption / generation capacity" and so on.
>
>     2.) Upon each timeslot (or maybe each day) change "aggregated"
>     statics about historic consumption and production of all customers
>     is made available to everybody (and we thought to keep them on a per
>     customer basis, which gives brokers a clearer picture on which
>     customer consumed / produced how much). For the statistics data we
>     think of boxplot information, i.e. mean, median, std, 25% 75%
>     quantile. Unclear is on how much history to aggregate (only one day
>     or maybe better a week or "type days").
>
>     3.) After timeslot deactivation each customer sends out his "meter
>     reading record", which is stored on the server in the accounting
>     service, and forwarded to the distribution utility for computing
>     balancing power. Also the respective brokers get "their" meter
>     readings, i.e. those from their customers.
>
>     So how does that match to your suggestions?
>
>         It's been argued that is it unrealistic to expect such
>         information to be available at the granularity of customers or
>         customer "clusters" defined in any useful way. That's probably
>         true. But that still leaves the question of exactly what
>         information brokers should be able to see or purchase.
>
>     That's right. So what we call "customer master data" should be
>     really a very rough categorization only (brokers should be able to
>     learn that a customer is a producer, for example, and what is
>     technical wind-to-energy ratio would be).
>
>         - Historical consumption and production data is available at any
>         time, including at the beginning of the game, for the entire
>         population. This data is broken down into hourly amounts for
>         weekdays and weekends, for each month of the year. So for a
>         12-month year, you would get an array 12x2x24 for both
>         production and consumption.
>
>     This is a "more detailed" version of our "aggregate statistics"
>     idea. I think, both is well possible to realize technically. To me,
>     the question is how much data we really want to reveal. I can't
>     judge. We replace the "is available any time" with a "will be
>     updated and sent out to brokers every hour (or maybe every day)".
>
>         - Actual consumption and production data, aggregated over the
>         entire population, is published at the end of each timeslot.
>         This will allow brokers to build up their own models and compare
>         them with the aggregate historical data.
>
>     We didn't think of overall aggregated consumption and production
>     data but probably that's a nice value add, in particular if we
>     decide to go with the minimal information solution and only send out
>     the "box plot statistics" for each customer.
>
>         - Actual consumption and production data is available to
>         individual brokers, at the end of each timeslot, aggregated over
>         each of their existing tariffs.
>
>     Here we thought, that they should really get the "raw meter
>     readings". I find that a bit more natural as this is the data they
>     get in reality too with smart meters installed. As said before:
>     Accounting Service stores all raw data for reference and
>     Distribution Utility gets aggregated data on a per broker basis so
>     that i can compute imbalances and impose balancing power fees.
>
>         - No predictions for weather-related production (solar and wind
>         sources) are available. Instead, the output of these sources
>         will be driven by the environment model, with an appropriate
>         level of added noise. Brokers must use weather forecasts along
>         with the published production functions to make their own
>         predictions.
>
>     Just to avoid misunderstanding: Weather forecasts != predictions?
>
>     We understand that you have access to historic weather data as well
>     as corresponding historic weather forecast data (= predictions?!),
>     right? Our idea was to send out the historic forecast data to
>     brokers so that they can set up their own forecasting models given
>     that they know the basic wind-to-power and sun-to-power ratios (or
>     even more elaborate "production functions" as you call it) of their
>     customers (they are published in publicly available customer master
>     data). The true weather data is only sent out  for the "now"
>     timeslot to (i) customers and (ii) brokers. Customer models can use
>     this data to compute their actual production (or maybe even
>     consumption if a consumer uses electric heating or cooling and thus
>     cares about outside temperature). Brokers get this data as feedback
>     for their prediction models.
>
>     Artificially generated demand / production forecasts are no longer
>     sent out.
>
>     To sum up: We mainly need to decide, what and how much "aggregate
>     historic" data should be publicly available.
>
>     Cheers,
>
>     Carsten
>
>     Am 06.12.2010 um 22:02 schrieb grampajohn [via Power TAC Developers]:
>
>
>
>     I've been looking through the server documentation, and I cannot
>     find anything about how brokers get information about expected power
>     production and consumption. We also have to think carefully about
>     the form and granularity of such information.
>
>     One of the original ideas about the MIS was that it would, for a
>     fee, break down the customer population by "preference clusters" and
>     be able to tell brokers about various features of their behavior,
>     such as consumption profiles and price elasticity. Brokers could
>     then build tariffs that somehow target those customer groups in
>     order to maximize the value of their portfolios.
>
>     It's been argued that is it unrealistic to expect such information
>     to be available at the granularity of customers or customer
>     "clusters" defined in any useful way. That's probably true. But that
>     still leaves the question of exactly what information brokers should
>     be able to see or purchase. Here's a proposal for a minimal first-cut:
>
>     - Historical consumption and production data is available at any
>     time, including at the beginning of the game, for the entire
>     population. This data is broken down into hourly amounts for
>     weekdays and weekends, for each month of the year. So for a 12-month
>     year, you would get an array 12x2x24 for both production and
>     consumption.
>
>     - Actual consumption and production data, aggregated over the entire
>     population, is published at the end of each timeslot. This will
>     allow brokers to build up their own models and compare them with the
>     aggregate historical data.
>
>     - Actual consumption and production data is available to individual
>     brokers, at the end of each timeslot, aggregated over each of their
>     existing tariffs.
>
>     - No predictions for weather-related production (solar and wind
>     sources) are available. Instead, the output of these sources will be
>     driven by the environment model, with an appropriate level of added
>     noise. Brokers must use weather forecasts along with the published
>     production functions to make their own predictions.
>
>     Is this enough?
>
>     John
>
>     ------------------------------------------------------------------------
>
>     View message @
>     http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2029952.html
>     <http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2029952.html?by-user=t&by-user=t&by-user=t>
>     To start a new topic under Power TAC Developers, email [hidden
>     email] <http://user/SendEmail.jtp?type=node&node=2032533&i=0&by-user=t>
>     To unsubscribe from Power TAC Developers, click here
>     <
>
>
>     ------------------------------------------------------------------------
>
>     View message @
>    
http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2032533.html
>     <http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2032533.html?by-user=t&by-user=t>
>     To start a new topic under Power TAC Developers, email [hidden
>     email] <http://user/SendEmail.jtp?type=node&node=2033523&i=1>
>     To unsubscribe from Power TAC Developers, click here
>     <
>
>
>
>
>     ------------------------------------------------------------------------
>     View message @
>    
http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2033523.html
>     <http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2033523.html?by-user=t>
>
>     To start a new topic under Power TAC Developers, email [hidden
>     email] </user/SendEmail.jtp?type=node&node=2034337&i=1>
>     To unsubscribe from Power TAC Developers, click here
>     <
>
>
>
>
>
> ------------------------------------------------------------------------
> View message @
>
http://power-tac-developers.975333.n3.nabble.com/Source-form-and-granularity-of-consumption-production-records-and-predictions-tp2029952p2034337.html
> To start a new topic under Power TAC Developers, email
> [hidden email]
> To unsubscribe from Power TAC Developers, click here
> <
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Source, form, and granularity of consumption/production records and predictions

grampajohn
Administrator
In reply to this post by Prashant Reddy
On 12/07/10 10:13, ppr [via Power TAC Developers] wrote:

> Hi all,
>
> I've discussed this topic with John offline so I mostly agree with this
> proposal.  I have three comments/questions:
>
> 1) One outstanding question seems to be:  Is the historical data (i)
> aggregated over time, but available for each customer/customer-model
> separately or (ii) is it aggregated over all customers, but available
> for different periods (seasons?) separately, or (iii) is it aggregated
> over both dimensions?  Maybe there should be a fee for the breakdown by
> customer while the other two options are free?

I would prefer to keep it simple and somewhat realistic. Just overall
aggregate data, broken down by season, but not by customer. I've posted
a proposal for how to generate it at
https://github.com/powertac/domain-data/wiki/Consumption-and-Production-records-and-predictions
under "Generating customer data."

>
> 2) Free information published by a federal bureau is typically delayed
> by a few weeks or few months.  Do we want to offer the option of near
> real-time information for a fee?  I think the fee is useful if we think
> that every broker requesting or subscribing to real-time information
> will overwhelm the server/MarketDataService.

Let's see if we can get away with just the overall aggregate data, which
is intended to be representative of customer model behavior, but not
enough to avoid a certain level of sampling error.

>
> 3) Another outstanding question:  How is the initial historical data
> generated, self-declared or Chris' suggestion of -10-to-0 timeslots?  I
> liked Chris' idea initially, but thinking some more... if there is only
> one flat tariff initially, do we need the 10 timeslots?  Won't the
> customer models be able to deterministically generate their "historical
> consumption/production" if they simply know what that flat tariff is?
> Further, is the preexisting flat tariff known ahead of time to the
> customer models?  If so, then it seems that the customer models should
> be able to self-declare their historic data at start-up?

I expect customer models to be fairly stochastic as well as sensitive to
the environment, and of course the environment is stochastic. So I don't
think 10 timeslots would be very useful in developing the broker's model
of expected customer behavior. I think the broker would want hundreds of
timeslots over a range of environmental conditions.

John
Loading...