Broker identity in database and messages

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Broker identity in database and messages

grampajohn
Administrator

The term "broker" refers to two different, distinct entities in our current design; this seems wrong, and it is causing confusion. In the server, a "broker" is a data element, an instance of the class Broker, which is currently serializable, and for some reason that is unknown to me and not documented, it has three different unique identifiers. However, it's unclear why we would ever serialize this, because the "real" broker is a remote entity, not in the database, identified by its username.

Unless someone is able to provide a clear and compelling defense of the current design, here is what I propose:

  • Brokers are identified by username, both in the database and in data structures such as message channels. This means that the id and apiKey fields of the Broker class are not needed. GORM may get annoyed by this and add an id field if we don't have one. Does this matter?
  • Broker is NOT Serializable, nor is any type with a field of type Broker. There are many data elements that would be sucked up by serialization of a broker instance, and I see no compelling reason to write our own serializer to prevent this.
  • Serializable types do not include broker as a field, but brokerUsername instead. Otherwise we have to specifically control their serialization.
  • The Shout type is an example of a Serializable type that includes broker as a field, and it belongs to Broker (it also belongs to Product for some reason I don't understand). Perhaps Shout should be a wrapper for a Serializable type, as Tariff is a wrapper for TariffSpecification.

Does this make sense? If so, then some refactoring is needed. I have already done it (but not pushed yet) for TariffSpecification and all the related Serializable types.

John

Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer

The term "broker" refers to two different, distinct entities in our current design; this seems wrong, and it is causing confusion. In the server, a "broker" is a data element, an instance of the class Broker, which is currently serializable, and for some reason that is unknown to me and not documented, it has three different unique identifiers. However, it's unclear why we would ever serialize this, because the "real" broker is a remote entity, not in the database, identified by its username.

1. As you describe, the "Broker" class acts as a proxy for an outside broker within the server
2. Within the web-app, it's a class that also represents a broker, basically for identification by username and api key for the login process from broker to server *and* by username and password for logging into the web-app. This has not been worked out, but it might make sense to have it serializable in order for the web-app to communicate to the server which brokers participate in a competition. 

Unless someone is able to provide a clear and compelling defense of the current design, here is what I propose:

  • Brokers are identified by username, both in the database and in data structures such as message channels. This means that the id and apiKey fields of the Broker class are not needed. GORM may get annoyed by this and add an id field if we don't have one. Does this matter?
We need the API key in order to identify brokers that communicate with the server. We might not need the "id" field however. 
  • Broker is NOT Serializable, nor is any type with a field of type Broker. There are many data elements that would be sucked up by serialization of a broker instance, and I see no compelling reason to write our own serializer to prevent this.
Depending on the serializer we use, we can definitely control the depth of objects being serialized. So with the web-app to server communication in mind, I would say we should keep it serializable.

David
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
ddauer wrote
1. As you describe, the "Broker" class acts as a proxy for an outside broker within the server
2. Within the web-app, it's a class that also represents a broker, basically for identification by username and api key for the login process from broker to server *and* by username and password for logging into the web-app. This has not been worked out, but it might make sense to have it serializable in order for the web-app to communicate to the server which brokers participate in a competition.
You say the api key is needed for broker login, but that is the extent of documentation I have seen. What does this mean? I know conceptually what an api key is, but I don't understand how a broker agent acquires one, and what role it plays in our system. If we are using it, why do we also need a username, a password, and an internal broker ID? Is the API key something given to an agent team that uniquely identifies a particular agent? If so, then for identification purposes it seems equivalent to a username. Or is it generated by the web-app to signal to the server that the agent has a valid identity? This would make more sense.

I think having the web-app communicate the broker identities to the server as part of server initialization makes sense. We still have to work out whether that communication is through a communication channel or through a shared database. It might indeed be much simpler to use a communication channel for this - basically we would need a Competition instance and a set of Brokers.
ddauer wrote
We need the API key in order to identify brokers that communicate with the server. We might not need the "id" field however.
So you are saying that the api key is a valid, unique identifier in the context of a simulation? Then let's just call it the broker's ID within the context of the server.
ddauer wrote
Depending on the serializer we use, we can definitely control the depth of objects being serialized. So with the web-app to server communication in mind, I would say we should keep it serializable.
At the point where the web-app constructs and communicates a Broker instance to the server, all its associations would presumably be empty - no tariffs, no market positions, etc. So no special serialization would be needed. We just need to agree that the _server_ never serializes a Broker instance, and then we don't need a custom serializer.

This leaves open the question of how serializable types intended for server->broker communication will be serialized, and how they should identify the broker. Many of them have a field of type Broker. Some have a field of type String that contains a broker ID.

In order to nail this down, we need to settle on a serialization scheme. Are we going to use the Converters plugin, or did you have something else in mind? It appears that the plugin does shallow serialization by default, which means we can just use object references in our data model without worrying about serialization issues, as long as the ID values are what we want to have serialized. That would argue for using the api key as the database ID for brokers.

When I look at the DomainClassTransformer class, it seems to depend on this plugin, but I can find no reference to the plugin, and the recommended configuration is not in Config.groovy in  either common or server. Also, I don't understand much of what's in there. We are not using any of the joda time types that are declared there, and the rest of the code seems much more complicated than the examples shown in the Grails Converter docs. Was this an attempt to avoid using the plugin?

Thanks. I really want to get this settled quickly.

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer


On Mon, Apr 4, 2011 at 4:59 PM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
ddauer wrote:
1. As you describe, the "Broker" class acts as a proxy for an outside broker within the server
2. Within the web-app, it's a class that also represents a broker, basically for identification by username and api key for the login process from broker to server *and* by username and password for logging into the web-app. This has not been worked out, but it might make sense to have it serializable in order for the web-app to communicate to the server which brokers participate in a competition.
You say the api key is needed for broker login, but that is the extent of documentation I have seen. What does this mean? I know conceptually what an api key is, but I don't understand how a broker agent acquires one, and what role it plays in our system. If we are using it, why do we also need a username, a password, and an internal broker ID? Is the API key something given to an agent team that uniquely identifies a particular agent? If so, then for identification purposes it seems equivalent to a username. Or is it generated by the web-app to signal to the server that the agent has a valid identity? This would make more sense.

Right, the API key is used for identity validation in combination with the username. At least that was how it was done with TAC Energy. We also need a username for the web-app, because users wouldn't want to login online with a UUID as name - the same goes for the password. The internal broker id however seems obsolete.
 

I think having the web-app communicate the broker identities to the server as part of server initialization makes sense. We still have to work out whether that communication is through a communication channel or through a shared database. It might indeed be much simpler to use a communication channel for this - basically we would need a Competition instance and a set of Brokers.
ddauer wrote:
We need the API key in order to identify brokers that communicate with the server. We might not need the "id" field however.
So you are saying that the api key is a valid, unique identifier in the context of a simulation? Then let's just call it the broker's ID within the context of the server.

Sounds good.
 
At the point where the web-app constructs and communicates a Broker instance to the server, all its associations would presumably be empty - no tariffs, no market positions, etc. So no special serialization would be needed. We just need to agree that the _server_ never serializes a Broker instance, and then we don't need a custom serializer.

Right, but since there is no good deserializer included with Grails, the server would still need to include Castor/XStream or some other tool.
 
This leaves open the question of how serializable types intended for server->broker communication will be serialized, and how they should identify the broker. Many of them have a field of type Broker. Some have a field of type String that contains a broker ID.

In order to nail this down, we need to settle on a serialization scheme. Are we going to use the Converters plugin, or did you have something else in mind? It appears that the plugin does shallow serialization by default, which means we can just use object references in our data model without worrying about serialization issues, as long as the ID values are what we want to have serialized. That would argue for using the api key as the database ID for brokers.

Grails Converters does an ok job of serializing but not the reverse - you can get an XML parser and would have to create your objects based on that. This is actually what the DomainClassTransformer is supposed to do, but only for domain classes (so no messages/commands).
 
When I look at the DomainClassTransformer class, it seems to depend on this plugin, but I can find no reference to the plugin, and the recommended configuration is not in Config.groovy in  either common or server. Also, I don't understand much of what's in there. We are not using any of the joda time types that are declared there, and the rest of the code seems much more complicated than the examples shown in the Grails Converter docs. Was this an attempt to avoid using the plugin?

As mentioned above, it was supposed to reverse the serialization done by Grails Converters - not to replace it.
 
David
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
ddauer wrote
.. since there is no good deserializer included with Grails, the
server would still need to include Castor/XStream or some other tool.
I am encouraged by my experiments with the simple xml serialization framework. So far I am serializing TariffSpecifications with attached Rates, and it's not too bad. It requires some annotation, but it does not require an xsd file for each class, as I believe Castor and JAXB do. You can see the code in test/integration/.../TariffSerializationTests in the serialization branch of powertac-common.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
In reply to this post by ddauer
OK, I think we are stuck. The unique identifier that the (remote) broker and the server agree on is called an apiKey. Is it required that it be called that in the Broker class? Because I've run into a brick wall trying to get GORM to believe it's a valid ID. You get very mysterious errors, depending on which set of advice you try to get it to work - either you end up with multiple Brokers having the same API key (violates constraints, and just plain wrong), or you get hibernate complaining about duplicate column names.

So what I will do for the time being is to use a getter and setter to make the apiKey into a synonym for id. If that's not going to work, please let me know ASAP.

Thanks -

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
In reply to this post by grampajohn
grampajohn wrote
I am encouraged by my experiments with the simple xml serialization framework. So far I am serializing TariffSpecifications with attached Rates, and it's not too bad. It requires some annotation, but it does not require an xsd file for each class, as I believe Castor and JAXB do. You can see the code in test/integration/.../TariffSerializationTests in the serialization branch of powertac-common.
I have settled on XStream - it seems to work well, and it eliminates the problem simple xml has of needing to know the class of the thing you are deserializing before you can process it.. Some of the tariff types, along with a serialization test, are committed in powertac-common.

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
In reply to this post by grampajohn
grampajohn wrote
So what I will do for the time being is to use a getter and setter to make the apiKey into a synonym for id. If that's not going to work, please let me know ASAP.
Actually, it's a getter method, but not a setter method. This means that the apiKey must be known at the time the Broker instance is created, at which point it is used as the 'id' attribute of the new Broker instance. Is this acceptable? We could create the illusion that it's the 'apiKey' attribute by writing a custom constructor.

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer
If this seems to create so much problems, why not just get rid of apikey, use id instead while we still can?

David
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
On 04/06/2011 02:44 PM, ddauer [via Power TAC Developers] wrote:
> If this seems to create so much problems, why not just get rid of
> apikey, use id instead while we still can?

That's essentially what's implemented right now. We just have to make
sure that the Broker gets created AFTER we know what the API key is
supposed to be. So in other words, the Broker instance probably has to
be created by the same process that creates the key and communicates it
to the Broker. Does this make sense?

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer

That's essentially what's implemented right now. We just have to make
sure that the Broker gets created AFTER we know what the API key is
supposed to be. So in other words, the Broker instance probably has to
be created by the same process that creates the key and communicates it
to the Broker. Does this make sense?

Not sure I follow, the api key was just like the id field a random uuid which can automatically be created at Broker creation. Also, the key is not supposed to be communicated to brokers but rather shown in the web-app to the broker developer when logging into the web-app. Once the developer obtains the key (or now the id), he has to put it into his broker which allows brokers to connect to the web-app/server in the first place.

David
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
On 04/06/2011 02:56 PM, ddauer [via Power TAC Developers] wrote:
>
> Not sure I follow, the api key was just like the id field a random uuid
> which can automatically be created at Broker creation. Also, the key is
> not supposed to be communicated to brokers but rather shown in the
> web-app to the broker developer when logging into the web-app. Once the
> developer obtains the key (or now the id), he has to put it into his
> broker which allows brokers to connect to the web-app/server in the
> first place.

OK, but I'm afraid this won't work for tournament competition. Broker
agents need to be able to run unattended for days at a time in a
tournament. There will be no per-game opportunity to copy data from a
web screen to a broker.

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

Prashant Reddy
I will second the opinion that the process of copying the apiKey from the webapp to the broker configuration always seemed unnecessarily complicated to me.  Why not just let each broker agent generate their own UUID and communicate that to the server when loggin in?  We can just say in the game rules that they have to use the same UUID over all the rounds of the competition.

Thanks,
Prashant



Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

chris.flath
I agree this is the most sensible way to handle this.
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer
In reply to this post by grampajohn
OK, but I'm afraid this won't work for tournament competition. Broker 
agents need to be able to run unattended for days at a time in a
tournament. There will be no per-game opportunity to copy data from a
web screen to a broker.

John


I see. The other option would be to not use an API key at all and have the brokers log in with their username/password that is also valid for the web-app. The username/password could also be used as credentials for every communication.

David

Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
On 04/07/2011 03:43 AM, ddauer [via Power TAC Developers] wrote:

>     OK, but I'm afraid this won't work for tournament competition. Broker
>     agents need to be able to run unattended for days at a time in a
>     tournament. There will be no per-game opportunity to copy data from a
>     web screen to a broker.
>
>     John
>
>
> I see. The other option would be to not use an API key at all and have
> the brokers log in with their username/password that is also valid for
> the web-app. The username/password could also be used as credentials for
> every communication.

Yes, that could work, although it's potentially extra data in every
single message, and it may complicate the association of brokers to
queues. An alternative would be to have the server create the Broker
instance when a broker first logs in, and send back the Broker instance
or just its ID to use for subsequent messages.

Just a thought...

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

ddauer

On Thu, Apr 7, 2011 at 4:08 PM, grampajohn [via Power TAC Developers] <[hidden email]> wrote:
Yes, that could work, although it's potentially extra data in every 
single message, and it may complicate the association of brokers to
queues. An alternative would be to have the server create the Broker
instance when a broker first logs in, and send back the Broker instance
or just its ID to use for subsequent messages.

OK sounds good. How would we make sure that a broker would not request multiple logins? Would the broker logging in be identified by a unique username taken from the web-app?

David
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
On 04/07/2011 09:13 AM, ddauer [via Power TAC Developers] wrote:

>
> On Thu, Apr 7, 2011 at 4:08 PM, grampajohn [via Power TAC Developers]
> <[hidden email]
> </user/SendEmail.jtp?type=node&node=2790420&i=0&by-user=t>> wrote:
>
>     Yes, that could work, although it's potentially extra data in every
>     single message, and it may complicate the association of brokers to
>     queues. An alternative would be to have the server create the Broker
>     instance when a broker first logs in, and send back the Broker instance
>     or just its ID to use for subsequent messages.
>
>
> OK sounds good. How would we make sure that a broker would not request
> multiple logins? Would the broker logging in be identified by a unique
> username taken from the web-app?

In a tournament situation, broker usernames are pre-determined for each
game; only brokers with usernames and passwords matching those specific
registrations will be logged in. Note also that in a tournament
situation, most likely all the brokers in the tournament will attempt to
log in to each game - it's easier to design the brokers that way. We
just have to admit the ones who are scheduled for that particular game.

Does this help?

John
Reply | Threaded
Open this post in threaded view
|

Re: Broker identity in database and messages

grampajohn
Administrator
As implemented right now, the broker's username is used to identify the broker in messages passed between broker and server. This turned out to make the most sense, because the identities of the competing brokers is supposed to be revealed to all brokers at the beginning of the game.