Bundles waiting for JPATemplate service on Apache Karaf startup

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Bundles waiting for JPATemplate service on Apache Karaf startup

Dheeraj Guntupalli
Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.


Thanks & regards,

Dheeraj

Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Raymond Auge
I'm not sure if it's the case in Karaf, but isn't the "bundle cache" where Configuration Admin configurations are stored? I wouldn't delete the bundle cache unless you are prepared to recreate all the configurations that would be deleted each time.

Note that the "bundle cache" is also a persistence storage area for bundles. They are allowed to use this space to hold files that keep their state, which is how configuration admin works by default. And so when you delete the "bundle cache" you also delete configuration admin's persistent store, and all configurations. But there might also be other bundles that use this mechanism.

I hope that helps,
- Ray

On Wed, Dec 19, 2018 at 6:44 AM Dheeraj Guntupalli <[hidden email]> wrote:
Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.


Thanks & regards,

Dheeraj



--
Raymond Augé (@rotty3000)
Senior Software Architect Liferay, Inc. (@Liferay)
Board Member & EEG Co-Chair, OSGi Alliance (@OSGiAlliance)
Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Dheeraj Guntupalli
Thanks for the reply Ray.

In our application we have a monitoring bundle which deletes the bundle cache as part of self healing in some cases.

On restarting we might face this issue with JPATemplate, and hence we need a workaround or solution for it.

On our dev environment too we are able to see this issue frequently on deleting Karaf bundle cache

Thank you,
Dheeraj
Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

David Jencks-3
I can easily imagine you causing yourself problems like this if you delete the bundle cache while the framework is running. Deleting it while the framework is stopped should give you a clean startup but if it’s running very possibly some bundles will save stuff to the cache and others won’t, leaving you with an unworkable startup state.
David Jencks

Sent from my iPhone

> On Dec 20, 2018, at 12:31 AM, Dheeraj Guntupalli <[hidden email]> wrote:
>
> Thanks for the reply Ray.
>
> In our application we have a monitoring bundle which deletes the bundle cache as part of self healing in some cases.
>
> On restarting we might face this issue with JPATemplate, and hence we need a workaround or solution for it.
>
> On our dev environment too we are able to see this issue frequently on deleting Karaf bundle cache
>
> Thank you,
> Dheeraj
Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Christian Schneider
In reply to this post by Dheeraj Guntupalli
I also think this might be related to deleting the bundle cache while the system still runs. If you want a clean startup then you should delete the karaf data dir while the system is stopped. This also deletes the installed karaf features though. So to get your system up automatically you must configure your features as boot features. 

If you save all your configs in the etc dir then deleting the data dir will still preserve configuration.

Christian

Am Mi., 19. Dez. 2018 um 12:44 Uhr schrieb Dheeraj Guntupalli <[hidden email]>:
Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.


Thanks & regards,

Dheeraj



--
--
Christian Schneider

http://www.liquid-reality.de

Computer Scientist

Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Dheeraj Guntupalli
Hi,

Sorry for not being clear, we don't delete the bundle cache when karaf is up and running. We delete the cache after Karaf is stopped.

The self healing in our app is done by external program, and we didn't see any problem with that. Even in dev environment we delete cache manually and start karaf.

We see this issue only when Karaf is started with clean state.

Thanks & regards
Dheeraj

Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Jochen Walz
In reply to this post by Dheeraj Guntupalli

Dear all,

after possible misunderstandings, maybe some clarification for this scenario: "deleting the bundle cache" is done BEFORE starting Karaf in this case, to start from a clean status. Only after deleting the data folder, Karaf ist started.

When repeating this 10 times (starting Karaf from the "clean" state - i.e. the data folder deleted), ~ 5 times everything starts correctly. The other 5 times, the problem with missing JPATemplate services occurs. To be precise: typically, the JPATemplate service is only missing for two out of three persistence units.

When the system is either in the "failure" state or the "correct" state, restarting Karaf WITHOUT deleting the data folder before the new start, it stays in this state.

From this, we have drawn some conclusions:

1. The configuration (data sources, persistence bundles' manifest, persistence units) is basically correct, otherwise it would never start up properly.
2. It looks like somehow connected with some bundle start order, which is "memorized" in the bundle cache. As mentioned above, once the startup order is "correct", it will always start correctly when restarting Karaf without deletion of the data folder. When it is "wrong", we will always end up with missing JPATemplate services.

Additional information:

- The "raw" data sources are created via pax-jdbc-config. We can always see that they are registered correctly.
- The "derived" data sources are created after running Flyway migration scripts on the "raw" data sources. Also those are always registered correctly. The persistence units then use the "derived" data sources. Piping them through Flyway is not done with the ops4j.preHook, but by a component in a "DB migration" bundle.
- All application bundles - including the ones which call the Flyway migration - have no start-level specified. I.e. they all start with 80. When we still used Blueprint, we had seen some issues with this, but after migrating to Declarative Services, the resolution of the service dependencies always worked properly.
- It is interesting that with up to 2 persistence units, everything works well. When adding the third, the problem shows up now and then.
- Don't know whether this is relevant, but the typical scenario is to have 2 persistence units (and data sources) working on one database, different schemas, the third one working on another.
- We have seen this mainly for H2 databases, but sometimes also for PostgreSQL.

So currenty we don't have a clue why sometimes the JPATemplate services for two out of three persistence units doesn't show up in the service registry.

Regards,

Jochen


Am 19.12.2018 um 12:43 schrieb Dheeraj Guntupalli:
Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.


Thanks & regards,

Dheeraj

Reply | Threaded
Open this post in threaded view
|

Fwd: Bundles waiting for JPATemplate service on Apache Karaf startup

Dheeraj Guntupalli

Dear all

 

To further add to the discussion:

 

We observed that if component1 references this JpaTemplate service, and component2 references component1, and component2 activate method operation takes time, we see that JpaTemplate for the remaining two persistent units are not registered. This occurs 4 out of 10 times.

 

When we removed this operation inside component2 activate method, we are able to see the JpaTemplate services registered for all three persistent units.

 

So it appears that the bundle startup order/timing has an impact on provisioning of the JpaTemplate services – which it shouldn’t. Is this observed by others too? And how to resolve this?

 

Thanks & Regards,

Dheeraj

 

Dear all,

after possible misunderstandings, maybe some clarification for this scenario: "deleting the bundle cache" is done BEFORE starting Karaf in this case, to start from a clean status. Only after deleting the data folder, Karaf ist started.

When repeating this 10 times (starting Karaf from the "clean" state - i.e. the data folder deleted), ~ 5 times everything starts correctly. The other 5 times, the problem with missing JPATemplate services occurs. To be precise: typically, the JPATemplate service is only missing for two out of three persistence units.

When the system is either in the "failure" state or the "correct" state, restarting Karaf WITHOUT deleting the data folder before the new start, it stays in this state.

From this, we have drawn some conclusions:

1. The configuration (data sources, persistence bundles' manifest, persistence units) is basically correct, otherwise it would never start up properly.
2. It looks like somehow connected with some bundle start order, which is "memorized" in the bundle cache. As mentioned above, once the startup order is "correct", it will always start correctly when restarting Karaf without deletion of the data folder. When it is "wrong", we will always end up with missing JPATemplate services.

Additional information:

- The "raw" data sources are created via pax-jdbc-config. We can always see that they are registered correctly.
- The "derived" data sources are created after running Flyway migration scripts on the "raw" data sources. Also those are always registered correctly. The persistence units then use the "derived" data sources. Piping them through Flyway is not done with the ops4j.preHook, but by a component in a "DB migration" bundle.
- All application bundles - including the ones which call the Flyway migration - have no start-level specified. I.e. they all start with 80. When we still used Blueprint, we had seen some issues with this, but after migrating to Declarative Services, the resolution of the service dependencies always worked properly.
- It is interesting that with up to 2 persistence units, everything works well. When adding the third, the problem shows up now and then.
- Don't know whether this is relevant, but the typical scenario is to have 2 persistence units (and data sources) working on one database, different schemas, the third one working on another.
- We have seen this mainly for H2 databases, but sometimes also for PostgreSQL.

So currenty we don't have a clue why sometimes the JPATemplate services for two out of three persistence units doesn't show up in the service registry.

Regards,

Jochen

 

Am 19.12.2018 um 12:43 schrieb Dheeraj Guntupalli:

Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.

 

Thanks & regards,

Dheeraj



This transmission is intended solely for the addressee and contains confidential information.
If you are not the intended recipient, please immediately inform the sender and delete the message and any attachments from your system.
Furthermore, please do not copy the message or disclose the contents to anyone unless agreed otherwise. To the extent permitted by law we shall in no way be liable for any damages, whatever their nature, arising out of transmission failures, viruses, external influence, delays and the like.

Reply | Threaded
Open this post in threaded view
|

Re: Bundles waiting for JPATemplate service on Apache Karaf startup

Dheeraj Guntupalli
In reply to this post by Jochen Walz
Dear All,

To further add we observed that if component 1 references this JpaTemplate service, and component2 references component 1 , and  component 2 activate method operation takes time, we see that the JpaTemplate for the remaining two persistent units are not registered. this occurs 4 out of 10 times.

When we removed this operation inside component 2 activate method, we are able to see the JpaTemplate service s registered for all three persistence units.

So it appears that the bundle startup order/timing has an impact on provisioning of the JpaTemplate services - which it shouldn't. Is this observed by others too? And any ideas to avoid this issue?

Thanks and regards
Dheeraj

On Sat 22 Dec, 2018, 5:49 PM Jochen Walz, <[hidden email]> wrote:

Dear all,

after possible misunderstandings, maybe some clarification for this scenario: "deleting the bundle cache" is done BEFORE starting Karaf in this case, to start from a clean status. Only after deleting the data folder, Karaf ist started.

When repeating this 10 times (starting Karaf from the "clean" state - i.e. the data folder deleted), ~ 5 times everything starts correctly. The other 5 times, the problem with missing JPATemplate services occurs. To be precise: typically, the JPATemplate service is only missing for two out of three persistence units.

When the system is either in the "failure" state or the "correct" state, restarting Karaf WITHOUT deleting the data folder before the new start, it stays in this state.

From this, we have drawn some conclusions:

1. The configuration (data sources, persistence bundles' manifest, persistence units) is basically correct, otherwise it would never start up properly.
2. It looks like somehow connected with some bundle start order, which is "memorized" in the bundle cache. As mentioned above, once the startup order is "correct", it will always start correctly when restarting Karaf without deletion of the data folder. When it is "wrong", we will always end up with missing JPATemplate services.

Additional information:

- The "raw" data sources are created via pax-jdbc-config. We can always see that they are registered correctly.
- The "derived" data sources are created after running Flyway migration scripts on the "raw" data sources. Also those are always registered correctly. The persistence units then use the "derived" data sources. Piping them through Flyway is not done with the ops4j.preHook, but by a component in a "DB migration" bundle.
- All application bundles - including the ones which call the Flyway migration - have no start-level specified. I.e. they all start with 80. When we still used Blueprint, we had seen some issues with this, but after migrating to Declarative Services, the resolution of the service dependencies always worked properly.
- It is interesting that with up to 2 persistence units, everything works well. When adding the third, the problem shows up now and then.
- Don't know whether this is relevant, but the typical scenario is to have 2 persistence units (and data sources) working on one database, different schemas, the third one working on another.
- We have seen this mainly for H2 databases, but sometimes also for PostgreSQL.

So currenty we don't have a clue why sometimes the JPATemplate services for two out of three persistence units doesn't show up in the service registry.

Regards,

Jochen


Am 19.12.2018 um 12:43 schrieb Dheeraj Guntupalli:
Hello All,

We are using Karaf 4.1.7, aries-jpa 2.7.0, h2 db in our application. We use multiple persistence units, multiple h2 db files within our application. We observed that when we have multiple h2 db files, and when we restart Karaf after deleting the bundle cache a couple or more times we see this issue "Waiting for JpaTemplate".

We are able to reproduce this issue frequently, and when this issue occurs, the JPA Template services are missing in the OSGi service registry. We feel it is related to bundle/service startup/registration timing. Once the bundle cache is built and it works, it will also work on subsequent startups as long as the bundle cache isn’t deleted.

The persistence bundle creates the JpaTemplate service by providing the persistence unit, but also consumes it in the DAO services. Possibly the required waiting time before the input data sources become available can trigger the issue.

 Has anyone faced this issue earlier, or could give any pointers on how to resolve this. Please let us know if you need more info.


Thanks & regards,

Dheeraj