Updated 1.2.0-SNAPSHOT deployed today

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Greetings -

In preparation for the upcoming tournament season, I have deployed a fresh update in the 1.2.0-SNAPSHOT series. The most noticeable feature of this new snapshot is the inclusion of the electric-vehicle and forklift-truck models, making considerably more storage capacity available for balancing. There are also several minor bugfixes and some improvements to the state log. For example, TariffSubscriptions are now logged, and RegulationCapacity instances now include the ids of their respective subscriptions. This should make it much easier to see what's happening with balancing resources.

Some of the models are now creating tariff-specific usage profiles, addressing the main problem in Issue #795. We will try to expand this set in the next couple of weeks.

Tariff evaluation now does a better job of accounting for early-withdrawal fees as described in Issue #807, but there's a bit more to do in this area. Daniel suggests an approach that computes tariff dominance per-customer (using the customer's inconvenience-weight values), but I feel that this may slightly violate the imperfect rationality principle. I have been working on an absolute tariff-dominance scheme, but that's a bit more complicated and it's not finished. Any suggestions in this area would be welcome.

Please let us know ASAP if there are specific issues you would like to see addressed before this years tournament. I hope to do a formal 1.2.0 release prior to the qualifying round.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

David Lau
Dear John, thanks for improving the server. I'm David, one of the participant of PowerTac 2015. I just tried the latest release and found out some abnormal scenario:

1) The total customer count is normally 50k. But during my testing, the customer count will grow, and suddenly my broker's customer count could exceed 300k and some other testing competing brokers could reach 150k as well.

2) Some brokers could able to reach an enormous 125k profit in single timeslot.

My testing setting runs perfectly fine for the previous release. I'm not sure what kind of logs I should provide to you to investigate on those problems, maybe you can ping me on that so I can send you the appropriate data for analysis.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Hello, David -

This indeed sounds like a problem. Could you perhaps zip or gzip up the trace and state logs from a game that shows the problem, and post them somewhere temporarily so I can download them? They are far too big for email attachments or even for dropbox. I should have time today to look at them.

Thanks.

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
In reply to this post by David Lau
Hello, David -

David Lau wrote
...
1) The total customer count is normally 50k. But during my testing, the customer count will grow, and suddenly my broker's customer count could exceed 300k and some other testing competing brokers could reach 150k as well.
...
Where are you getting these numbers? From the Visualizer? By counting SIGNUP and WITHDRAW transactions (that's what the Visualizer does)? If so, the result is not correct in the face of tariff revocations. This is due to the fix about a year ago for Issue #762. So I guess we have two choices: (1) revisit the fix for #762, perhaps by emitting the WITHDRAW transaction without the early-withdrawal fee, or (2) fix the Visualizer to account for revoked tariffs correctly.

I've checked through the numbers in a case with revoked tariffs, and indeed the number of customers remains stable, but the numbers of SIGNUP and WITHDRAW transactions do not match.

Any suggestions? The critical bit of code is in TariffSubscription around line 218.

Cheers -

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
In reply to this post by David Lau
David and all -

I have updated the handling of subscription changes in response to revoked tariffs, and re-deployed the 1.2.0-SNAPSHOT package with the update. The "cumulative score" display for the Tariff Market in the Visualizer is now showing the correct number of customers after tariff-revoke events. I doubt any brokers were depending on that transaction NOT arriving for a revoked tariff; if your design depended on this dubious feature, I apologize.

I also re-deployed the common module; it's possible that the previous deploy was slightly inconsistent due to the deployed version of common being slightly out-of-date.

Please let me know if the behavior now seems improved. If you see problems, I want to know.

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

David Lau
Dear John,

Yes, I checked the number on both the Visualizer and the count from Signup and Withdraw. I just tested your new update on snapshot, seems that the problem still occurred.

I had uploaded the log file of the server to Dropbox. Please kindly check it out:
https://www.dropbox.com/sh/9qssj7qc0nsg577/AAAmRNqAvAvJ2OIAu4mTlBgNa?dl=0

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

MDyshel
In reply to this post by grampajohn
Hi John!

Thanks for deploying the snapshot. Unfortunately, since the update, I can't manage to run the server from command line... I get a long exception starting with

Exception in thread "main" org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/a/fr-05/vol/prime/grad/smdyshel/.m2/repository/org/powertac/customer-models/1.2.0-SNAPSHOT/customer-models-1.2.0-SNAPSHOT.jar!/org/powertac/common/TariffEvaluationHelper.class]; nested exception is java.io.EOFException: Unexpected end of ZLIB input stream


I tried to re-download the server, but it didn't help.

Also, I tried to clone the development repository instead of the zip releases I was using so far, but there's a problem with cloning the evcustomer submodule:

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of 'git@github.com:powertac/evcustomer.git' into submodule path 'evcustomer' failed

Is there anything I can do about it?
Thank you,
Maria

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Thanks. This may explain a problem another intrepid new user is having. I am testing a fresh download now.

On the ev-customer module - I'm not sure why this is happening, but I had the same problem on my laptop, just with ev-customer, until I uploaded a new ssh key. I think perhaps the problem has to do with a recent update of ssh on the client end. If this solves the problem for you, let me know and I'll make an announcement.

I'll keep you posted on the snapshot.

John


On 03/25/2015 09:46 AM, MDyshel [via Power TAC Developers] wrote:
Hi John!

Thanks for deploying the snapshot. Unfortunately, since the update, I can't manage to run the server from command line... I get a long exception starting with

Exception in thread "main" org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/a/fr-05/vol/prime/grad/smdyshel/.m2/repository/org/powertac/customer-models/1.2.0-SNAPSHOT/customer-models-1.2.0-SNAPSHOT.jar!/org/powertac/common/TariffEvaluationHelper.class]; nested exception is java.io.EOFException: Unexpected end of ZLIB input stream


I tried to re-download the server, but it didn't help.

Also, I tried to clone the development repository instead of the zip releases I was using so far, but there's a problem with cloning the evcustomer submodule:

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of '[hidden email]' into submodule path 'evcustomer' failed

Is there anything I can do about it?
Thank you,
Maria




If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/Updated-1-2-0-SNAPSHOT-deployed-today-tp4025894p4025909.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
In reply to this post by MDyshel
Hi, Maria -

On 03/25/2015 09:46 AM, MDyshel [via Power TAC Developers] wrote:
Hi John!

Thanks for deploying the snapshot. Unfortunately, since the update, I can't manage to run the server from command line... I get a long exception starting with

Exception in thread "main" org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: URL [jar:file:/a/fr-05/vol/prime/grad/smdyshel/.m2/repository/org/powertac/customer-models/1.2.0-SNAPSHOT/customer-models-1.2.0-SNAPSHOT.jar!/org/powertac/common/TariffEvaluationHelper.class]; nested exception is java.io.EOFException: Unexpected end of ZLIB input stream

This is very strange. I just pulled down the latest server-distribution 1.2.0-SNAPSHOT and stuck it in a test directory, isolated from my development environment. I ran a boot using the cli, no problems. I then ran mvn -Pweb and ran a game, then shut that down and ran a game with mvn -Pcli and the Sample broker. It all works OK for me.

One thing to try would be to clear out your local maven repo ~/.m2/repository/org/powertac  and force it to re-load all the components.

Thanks for your clear description of the problem Let me know if it persists.

John




I tried to re-download the server, but it didn't help.

Also, I tried to clone the development repository instead of the zip releases I was using so far, but there's a problem with cloning the evcustomer submodule:

Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of '[hidden email]' into submodule path 'evcustomer' failed

Is there anything I can do about it?
Thank you,
Maria




If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/Updated-1-2-0-SNAPSHOT-deployed-today-tp4025894p4025909.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
In reply to this post by grampajohn
Dear colleagues -

Today I re-deployed the server 1.2.0-SNAPSHOT. This version fixes Issue #807, which I suspect was causing at least some "interesting" behavior a few of you have reported recently. It's clear that #807 was a critical bug; I would be very interested to hear whether this update is a noticeable improvement.

Thanks.

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

serkan
I have just tested and works perfectly. Thank you for the quick fix.

Perhaps, a new trial tournament would be nice for a proper test. Personally, I couldn't test AgentUDE due to customer count problem.
Reply | Threaded
Open this post in threaded view
|

RE: Updated 1.2.0-SNAPSHOT deployed today

Wolf
Administrator

Serkan and all,

 

We are currently looking into if we can setup a new short trial tournament next week.

 

Be back shortly!

Wolf

 

From: serkan [via Power TAC Developers] [mailto:ml-node+[hidden email]]
Sent: vrijdag 27 maart 2015 2:01
To: Wolf Ketter
Subject: Re: Updated 1.2.0-SNAPSHOT deployed today

 

I have just tested and works perfectly. Thank you for the quick fix.

Perhaps, a new trial tournament would be nice for a proper test. Personally, I couldn't test AgentUDE due to customer count problem.


If you reply to this email, your message will be added to the discussion below:

http://power-tac-developers.975333.n3.nabble.com/Updated-1-2-0-SNAPSHOT-deployed-today-tp4025894p4025913.html

To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

MDyshel
In reply to this post by grampajohn
Hi John,

Adding the ssh key indeed helped me to pull the full git tree - thanks for the tip (though I still failed to compile it).

As about the distribution version - I emptied my powertac folder in ~/.m2/, but I still can't manage to run the server. When I try to run the web interface with mvn -P web, there must be some problem with the visualizer, because I get a mistake:

2015-03-31 15:49:07.067:WARN:oejw.WebAppContext:Failed startup of context o.m.j.p.JettyWebAppContext{/visualizer,[file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/visualizer/, file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/tmp/visualizer-1_2_0-SNAPSHOT_war/]},file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/visualizer/
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
        at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
        at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
        at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
        at org.eclipse.jetty.webapp.JarScanner.matched(JarScanner.java:162)
        at org.eclipse.jetty.util.PatternMatcher.matchPatterns(PatternMatcher.java:100)

... and the exception goes on and on. At the end, I have the message 'Started Jetty Server', but can't access the visualizer in my browser. Cleaning the target or the repository again doesn't help. Maybe it's somehow connected to the maven jetty packages, but I'm a bit afraid to erase them, because for now I at least have a 1.1.0-version server that works, and I wouldn't want to lose it a few weeks before the tournament...

When I try to run the server from the command line in bootstrap mode, I have some problem connected with jOptimizer:

Exception in thread "Thread-2" java.lang.NoClassDefFoundError:com/joptimizer/optimizers/LPOptimizationRequest
        at org.powertac.customer.model.LiftTruck$LpPlan.solve(LiftTruck.java:1430)

Caused by: java.lang.ClassNotFoundException: com.joptimizer.optimizers.LPOptimizationRequest
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)



Guess I would just have to work with the old version for now... thank you a lot for your help anyway.
Maria
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Thanks, Maria -

I will try to find time to look at this later today.

John


On 03/31/2015 08:03 AM, MDyshel [via Power TAC Developers] wrote:
Hi John,

Adding the ssh key indeed helped me to pull the full git tree - thanks for the tip (though I still failed to compile it).

As about the distribution version - I emptied my powertac folder in ~/.m2/, but I still can't manage to run the server. When I try to run the web interface with mvn -P web, there must be some problem with the visualizer, because I get a mistake:

2015-03-31 15:49:07.067:WARN:oejw.WebAppContext:Failed startup of context o.m.j.p.JettyWebAppContext{/visualizer,[file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/visualizer/, file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/tmp/visualizer-1_2_0-SNAPSHOT_war/]},file:/a/store-04/h/lab/ai/smdyshel/workspace-sts/server-new/target/visualizer/
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
        at java.util.zip.ZipInputStream.read(ZipInputStream.java:194)
        at java.util.jar.JarInputStream.read(JarInputStream.java:207)
        at java.util.zip.ZipInputStream.closeEntry(ZipInputStream.java:140)
        at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:118)
        at java.util.jar.JarInputStream.getNextEntry(JarInputStream.java:142)
        at java.util.jar.JarInputStream.getNextJarEntry(JarInputStream.java:179)
        at org.eclipse.jetty.webapp.JarScanner.matched(JarScanner.java:162)
        at org.eclipse.jetty.util.PatternMatcher.matchPatterns(PatternMatcher.java:100)

... and the exception goes on and on. At the end, I have the message 'Started Jetty Server', but can't access the visualizer in my browser. Cleaning the target or the repository again doesn't help. Maybe it's somehow connected to the maven jetty packages, but I'm a bit afraid to erase them, because for now I at least have a 1.1.0-version server that works, and I wouldn't want to lose it a few weeks before the tournament...

When I try to run the server from the command line in bootstrap mode, I have some problem connected with jOptimizer:

Exception in thread "Thread-2" java.lang.NoClassDefFoundError:com/joptimizer/optimizers/LPOptimizationRequest
        at org.powertac.customer.model.LiftTruck$LpPlan.solve(LiftTruck.java:1430)

Caused by: java.lang.ClassNotFoundException: com.joptimizer.optimizers.LPOptimizationRequest
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)



Guess I would just have to work with the old version for now... thank you a lot for your help anyway.
Maria


If you reply to this email, your message will be added to the discussion below:
http://power-tac-developers.975333.n3.nabble.com/Updated-1-2-0-SNAPSHOT-deployed-today-tp4025894p4025925.html
To start a new topic under Power TAC Developers, email [hidden email]
To unsubscribe from Power TAC Developers, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

MDyshel
In reply to this post by grampajohn
Hi John,

Forget the previous post - looks like I had some problems with jOptimizer packages, and it all sorted out, I have the server working properly. Thanks a lot!

Maria
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

Porag
In reply to this post by grampajohn
Dear John,

Could you please tell me from where I can download the 1.2.0 server SNAPSHOT? Is there any major difference between 1.1.1 and 1.2.0?  

Thanks in advance
Porag
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Porag wrote
Could you please tell me from where I can download the 1.2.0 server SNAPSHOT? Is there any major difference between 1.1.1 and 1.2.0?  
If you want to work in a server source environment, it's the HEAD of the master branch of each repo in github.

If you just want to run it, instructions and links are on the GettingStarted page, near the top in the paragraph that starts "If you would prefer to use development snapshots". The biggest change from 1.1.1 is a couple of new customer models, ev-customer (electric vehicles with V2G capabilities) and forklift-truck (lots of big lead-acid batteries, runs a daily shift schedule). Also, some customers are now doing a much better job of evaluating time-of-use tariffs.

The thing about snapshots is there is always just one - the latest - and they change when I decide to do a re-deploy. I always make an announcement when I do that, but it sometimes means you have to adapt a bit. On the other hand, the 1.2.0-SNAPSHOT version is what's running in the trial runs right now, and I hope to do a 1.2.0 release early next week, prior to the start of the qualifying round. After that, the development snapshot will switch to 1.3.0-SNAPSHOT.

Does this help?

John
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

Porag
Yes it helped John. Thanks for your reply.

Regards
Porag
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

serkan
In reply to this post by grampajohn
I have an observation regarding the contract length.

I have seen some tariffs in the retail market which have very long contract length and zero early withdrawal p. and bonus payment.

These tariffs are able to attract customers normally as if they are not binding. However their customers can not leave the tariff even though I offer way cheaper tariffs.

I think customers should not hesitate to leave the tariff since the early withdrawal payment is null.

Is there any problem here or am I missing something?
Reply | Threaded
Open this post in threaded view
|

Re: Updated 1.2.0-SNAPSHOT deployed today

grampajohn
Administrator
Hello, Serkan -
serkan wrote
I have seen some tariffs in the retail market which have very long contract length and zero early withdrawal p. and bonus payment.

These tariffs are able to attract customers normally as if they are not binding. However their customers can not leave the tariff even though I offer way cheaper tariffs.

I think customers should not hesitate to leave the tariff since the early withdrawal payment is null.

Is there any problem here or am I missing something?
First, I assume this is not new behavior. You should re-read the discussion of "inertia" in Section 4.1 of the specification. The residential customers have inertia values ranging from around 70% to over 90%, which means that 90% of the population will not bother to evaluate new tariffs at a particular publication interval. Multiply that by a population of 30000 (Brookside Homes) and you can see that somewhere between 21000 and 27000 of those customer will NOT evaluate tariffs in any given cycle. As a result, it takes some time for a superior tariff to spread through the population.

If you believe there is a problem aside from this effect, please share your evidence. I am not aware of a problem at this point.

Cheers -

John
12