Deserialize xml from log messages

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Deserialize xml from log messages

mblum
Dear PowerTac Developers,

in order to test my broker, I want to send messages from logfiles to my "Manager" classes. I tried deserializing XML messages from the log using org.powertac.common.XMLMessageConverter, which works for some messages.

However, I can't deserialize "weather-report" for instance. Am I missing something?

Is there a better way to test modules in isolation, without having to run a server?

Thanks,
Manuel

Exception in thread "main" com.thoughtworks.xstream.converters.ConversionException: null : null
---- Debugging information ----
cause-exception     : java.lang.NullPointerException
cause-message       : null
class               : org.powertac.common.WeatherReport
required-type       : org.powertac.common.WeatherReport
converter-type      : com.thoughtworks.xstream.converters.reflection.ReflectionConverter
path                : /weather-report
line number         : 1
version             : null
-------------------------------
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1035)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1019)
        at com.thoughtworks.xstream.XStream.fromXML(XStream.java:895)
        at com.thoughtworks.xstream.XStream.fromXML(XStream.java:886)
        at org.powertac.common.XMLMessageConverter.fromXML(XMLMessageConverter.java:94)
        at de.unifreiburg.powertac.mllbroker.PortfolioManagerTest.replay(PortfolioManagerTest.java:33)
        at de.unifreiburg.powertac.mllbroker.PortfolioManagerTest.main(PortfolioManagerTest.java:45)
Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

grampajohn
Administrator
Manuel -

It's a little hard to know exactly what's going on here. Which log are the weather-report records coming from? A broker log? Many of the message types require that the current state be correct when they are received, but I'm not sure about weather reports. It's possible this could happen if the current time is incorrect.

It should be possible to use data from server state logs; the logtool package includes code to extract objects from a state log, and some examples that show how to extract exactly what you want. See https://github.com/powertac/powertac-tools for details.

We normally use mockito for unit testing in the server. You can find some simple examples in server-main/src/test/java/.../BrokerProxyServiceTest, and a somewhat more elaborate example in default-broker/src/test/java/.../DefaultBrokerServiceTest. The documentation at mockito.org is quite good.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

mblum
John, thanks for your answer.

The weather-report records come from the bootstrap data of the broker logs.
I'll have a look on the powertac-tools.

Manuel
Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

grampajohn
Administrator
On 03/14/2013 05:40 AM, mblum [via Power TAC Developers] wrote:
> John, thanks for your answer.
>
> The weather-report records come from the bootstrap data of the broker logs.
> I'll have a look on the powertac-tools.

I guess I don't quite understand what your setup is. There is a unit
test in common that deserializes weather reports, and the sample broker
receives them without problems, although it does not use them.

Could you please take a look at
common.WeatherReportTest.xmlSerializationTest() and see if your code is
substantially different.

Thanks.

John


Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

mblum
I am using exactly the same code as in common.WeatherReportTest.xmlSerializationTest(). I works fine, as long as the timeslot is null.

If not, I'm getting the above error.
Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

grampajohn
Administrator
On 03/14/2013 11:53 AM, mblum [via Power TAC Developers] wrote:
> I am using exactly the same code as in
> common.WeatherReportTest.xmlSerializationTest(). I works fine, as long
> as the timeslot is null.

OK, that's the key. The code that's failing is almost certainly in
common.xml.TimeslotConverter. That only works if there's a functioning
timeslotRepo, and it expects to get it from the
SpringApplicationContext. That makes it hard to write simple unit tests.

I have created issue #675 to address this. It will be a quick fix, but I
am reluctant to push out a change that would affect sample-broker this
close to a scheduled competition. Would it be acceptable to wait until
after 3/22?

Thanks.

John



Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

mblum
Yeah, sure. Thanks for your help.

Manuel
Reply | Threaded
Open this post in threaded view
|

Re: Deserialize xml from log messages

grampajohn
Administrator
In reply to this post by grampajohn
Colleagues -

Today I pushed a set of changes that replaces timeslot with timeslot serial number for all domain types. The result looks the same in the xml files, but they should now deserialize correctly without having Spring, TimeService, and TimeslotRepo set up and running. All the domain types that have a getTimeslot() method now also have a getTimeslotIndex() method that returns the serial number. If you really want the timeslot, it's available, but only if you are running in a Spring environment.

The changes are backward-compatible, so you should not have to edit your brokers, but you will have to re-compile once this change is deployed to the snapshot repo. I will hold off on that for several days and let those working in the server source environment a chance to pull down the updates and make sure they do not experience problems. Note that if you have not pulled down changes recently, you may get some Eclipse errors when you try to refresh a few of the modules. This is because I've been cleaning up project config files and updating the .gitignore files. You should just have to do Maven->Update Project Configuration to recover.

Note that a few other modules had to change for this to work (including Accounting, Auctioneer, and Genco), but those changes are in test code and should not affect anyone.

Please let me know right away if you experience problems with these changes.

Thanks.

John