Revoking a tariff causes errors.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Revoking a tariff causes errors.

jdknijff
When I try to revoke a Tariff in Power Tac 0.4, the server get's the following exception:

2011-07-03 21:52:06,198 [pool-1-thread-1] ERROR events.PatchedDefaultFlushEventListener  - Could not synchronize database state with session
org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.powertac.common.Tariff#4]
        at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:1792)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2435)
        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2335)
        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2635)
        at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
        at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
        at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
        at org.codehaus.groovy.grails.orm.hibernate.events.PatchedDefaultFlushEventListener.performExecutions(PatchedDefaultFlushEventListener.java:46)
        at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
        at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027)
        at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:365)
        at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:137)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        at org.powertac.accountingservice.AccountingService$$EnhancerByCGLIB$$5de8ede7.addTariffTransaction(<generated>)
        at org.powertac.common.interfaces.Accounting$addTariffTransaction.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
        at org.powertac.common.interfaces.Accounting$addTariffTransaction.call(Unknown Source)
        at org.powertac.common.TariffSubscription.usePower(TariffSubscription.groovy:197)
        at org.powertac.common.TariffSubscription$usePower.call(Unknown Source)
        at org.powertac.common.AbstractCustomer$_consumePower_closure2.doCall(AbstractCustomer.groovy:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at groovy.lang.Closure.call(Closure.java:282)
        at groovy.lang.Closure.call(Closure.java:295)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1220)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1196)
        at org.codehaus.groovy.runtime.dgm$110.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:270)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
        at org.powertac.common.AbstractCustomer.consumePower(AbstractCustomer.groovy:153)
        at org.powertac.common.AbstractCustomer$consumePower.callCurrent(Unknown Source)
        at org.powertac.common.AbstractCustomer.step(AbstractCustomer.groovy:471)
        at org.powertac.consumers.Village.super$2$step(Village.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:127)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:147)
        at org.powertac.consumers.Village.step(Village.groovy:606)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:793)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:776)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:163)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodNSafe(ScriptBytecodeAdapter.java:171)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodNSpreadSafe(ScriptBytecodeAdapter.java:178)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethod0SpreadSafe(ScriptBytecodeAdapter.java:197)
        at org.powertac.abstractcustomerservice.AbstractCustomerService.activate(AbstractCustomerService.groovy:101)
        at sun.reflect.GeneratedMethodAccessor1143.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:793)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:776)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:163)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodNSafe(ScriptBytecodeAdapter.java:171)
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodNSpreadSafe(ScriptBytecodeAdapter.java:178)
        at org.powertac.server.CompetitionControlService$_step_closure4.doCall(CompetitionControlService.groovy:223)
        at sun.reflect.GeneratedMethodAccessor1197.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at groovy.lang.Closure.call(Closure.java:282)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.eachWithIndex(DefaultGroovyMethods.java:1213)
        at org.codehaus.groovy.runtime.dgm$150.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:270)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
        at org.powertac.server.CompetitionControlService.step(CompetitionControlService.groovy:221)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1003)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
        at org.powertac.server.CompetitionControlService$_start_closure3.doCall(CompetitionControlService.groovy:194)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:225)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:51)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)
        at org.powertac.server.CompetitionControlService$_start_closure3.doCall(CompetitionControlService.groovy)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
        at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1070)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
        at groovy.lang.Closure.call(Closure.java:282)
        at groovy.lang.Closure.call(Closure.java:277)
        at groovy.lang.Closure.run(Closure.java:360)
        at java_lang_Runnable$run.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at grails.plugin.executor.SessionBoundRunnable.run(SessionBoundRunnable.groovy:39)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


After this error, the server sends a SimPause message to the broker and it doesn't seem to send a SimResume message after that. I've checked the PowerTac server and the tariff is revoked on it, but the simulation seems to have been stopped. Does anyone know how to fix this? I did not have this problem in Power Tac 0.3.
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

Markus
I see similar symptoms when submitting new Shouts. In my case both, the 0.3 and 0.4 release, are affected. I posted specifics and code to reproduce here: https://github.com/powertac/powertac-server/issues/306.

Anyone know how to fix this?

Thanks,

Markus
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

grampajohn
Administrator
In reply to this post by jdknijff
With your broker code, I can duplicate this error. I suspect we have a problem with interaction between the new threading model and the hibernate locking scheme. It's going to take some time to track it down.
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

grampajohn
Administrator
In reply to this post by Markus
As many of you know by now, it's not just revoking tariffs that causes errors. It's submitting shouts, and many other interactions that involve database access in the scope of jms calls on the server. The problem is that we get optimistic locking failures, which is not unexpected in a concurrent application.

One solution would be to serialize database access, using pessimistic locking, but that is not supported for the in-memory database, and going to an external database is both unnecessary and a performance problem.

Another solution, proposed by Nguyen, is that we serialize database access in a different way, simply by queuing up all the incoming messages and processing them in the simulation thread. This will be a non-trivial change, but I think it's do-able in a day or two.

If you want to follow and contribute to the detailed discussion on this issue, please go to the github issue thread.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

chris.flath
If going to an external database would help solving these problems I can actually say that it is not very difficult and not noticeably/ significantly slower - actually I have some better performance results with respect to the caching effects exhibited in the degradation problem.

Find below a quick mySQL workthrough (adapted from German http://www.aulich.org/wplog/?p=179):

1. Download current mySQL for your system from http://www.mysql.com/
2. create and run a setup script:

    DROP DATABASE RC;  
    CREATE DATABASE RC;  
    USE RC;  
    DROP USER 'rcuser'@'localhost';  
    CREATE USER 'rcuser'@'localhost';  
    GRANT ALL PRIVILEGES ON RC.* TO 'rcuser'@'localhost';

3. Install the Connector/J from mySQL.com into powertac-server/lib
4. Modify powertac-server/grails-app/conf/Datasource.groovy:

    development {  
        dataSource {  
            dbCreate = "create-drop"  
            url = "jdbc:mysql://localhost/rc"  
               driverClassName = "com.mysql.jdbc.Driver"  
               username = "rcuser"  
        }  
    }  
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

grampajohn
Administrator
In reply to this post by grampajohn
Queuing of incoming shouts solves the problem; I am using the broker from Markus, and I'm several days into the simulation without a problem. I still have to work out serialization for tariff-related messages.

John
Reply | Threaded
Open this post in threaded view
|

Re: Revoking a tariff causes errors.

grampajohn
Administrator
I have completed updates to the tariff market and the wholesale market that serialize processing of incoming messages. These changes are pushed to both master and to release-0.4. A consequence of these changes is that server response to broker messages will be delayed until the next time messages of that type are processed by the simulation. On the other hand, the transaction-related exceptions appear to be gone.

You may manually retrieve the new 0.4 packages (powertac-server, powertac-accounting-service, powertac-auctioneer-pda), or just run the 0.4 download script again. The latter will replace all your sources, so don't do this if you have been modifying the server.

Please let me know if you encounter further problems.

John