AEM 6.5 Upgrade to 6.5.7 CFP Causing Unresponsive Instances | Community
Skip to main content
sdouglasmcSonova
Level 4
Dec 18, 2020 3:07 PM UTC
Solved

AEM 6.5 Upgrade to 6.5.7 CFP Causing Unresponsive Instances

We've recently upgraded AEM from 6.5.5 to 6.5.7 (didn't install 6.5.6) - running on Ubuntu 18.04.5, Java 11.

We have been pretty consistent the last 8-9 months after making some performance adjustments in migrating from 6.3 windows to 6.5 Unbuntu.  Rarely having to restart.  However, since we've upgraded from SP 6.5.5 to 6.5.7 (we bipassed 6.5.6 - which SHOULDN'T matter), we've noticed really poor performance, author instance needing restarts nearly every 2-3 days and publish instances 5-6 days because of becoming unresponsive.  No heap dumps or OutOfMemory exceptions.

I'm just wondering if anybody else might know of any obvious reasons this could be happening before I bog myself down in a sea of heapdumps and threaddumps.

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    NarayanaChAccepted solution
    Level 3
    Jan 18, 2021 9:08 PM UTC

    @sdouglasmcsonova we also got the same package from Adobe, it resolved our issue.

    Notes from adobe:

    Checking the thread dumps and further researching internally, It seems you are running into a known issue(CQ-4312194) with SP7 where numerous threads get blocked due to a Timer with the Component Registry (org.apache.felix.scr.impl.ComponentRegistry). This causes the instance to become unresponsive.

    • Follow the steps below to resolve the issue:
      - Install the attached hotfix package
      - This will trigger a restart of a couple of bundles. So, need to wait 3-5 mins
      - Go to <host>:<port>/system/console and make sure the "org.apache.felix.scr" version is updated to 2.1.20
    View reply

    32 replies

    bartek__w
    Level 2
    Jul 21, 2021 1:57 PM UTC

    Great article, thanks !

     

    Is this package available publicly now ? 

     

     

     
    Translate
      JakubLG
      Level 2
      Jan 20, 2021 1:48 PM UTC

      For those who haven't updated to 6.5.7 yet, Adobe has a new SP7 installation package with Hotfix-CQ-4312194-FELIX-6252 included in it. Makes upgrade faster and more stable. This is not publicly available (yet) but you can probably ask for it in a ticket.

      Translate
        kautuk_sahni
        kautuk_sahni
        Employee
        Jan 21, 2021 8:59 AM UTC
        @jakublg Thank you for sharing this with community. Keep assisting others here.
        Translate
        Kautuk Sahni
        NarayanaChAccepted solution
        Level 3
        Jan 18, 2021 9:08 PM UTC

        @sdouglasmcsonova we also got the same package from Adobe, it resolved our issue.

        Notes from adobe:

        Checking the thread dumps and further researching internally, It seems you are running into a known issue(CQ-4312194) with SP7 where numerous threads get blocked due to a Timer with the Component Registry (org.apache.felix.scr.impl.ComponentRegistry). This causes the instance to become unresponsive.

        • Follow the steps below to resolve the issue:
          - Install the attached hotfix package
          - This will trigger a restart of a couple of bundles. So, need to wait 3-5 mins
          - Go to <host>:<port>/system/console and make sure the "org.apache.felix.scr" version is updated to 2.1.20
        Translate
          kautuk_sahni
          kautuk_sahni
          Employee
          Jan 20, 2021 1:04 PM UTC
          Thank you for sharing this with community.
          Translate
          Kautuk Sahni
          NarayanaCh
          Level 3
          Jan 18, 2021 4:33 PM UTC

          @sdouglasmcsonova is there any resolution or solution on this issue? we are also facing the same issue in our Prod environment.

          Translate
            sdouglasmcSonovaAuthor
            Level 4
            Jan 18, 2021 5:16 PM UTC
            Get the Hotfix mentioned in the comments of this thread.
            Translate
              JakubLG
              Level 2
              Dec 27, 2020 9:06 AM UTC

              Can you check threaddumps. I've noticed AEM 6.5.7 hanging on following deadlock on my environments:

              java.lang.Thread.State: WAITING (on object monitor) at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method) - waiting to lock <0x1d24408f> (a java.util.concurrent.CountDownLatch$Sync) owned by "null" tid=0x-1 at java.base@11.0.9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) at java.base@11.0.9/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) at java.base@11.0.9/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:365)
              Translate
                sdouglasmcSonovaAuthor
                Level 4
                Dec 27, 2020 2:30 PM UTC
                They've sent me a Hotfix-CQ-4312194-FELIX-6252-1.0.zip. I've installed it but I'm not sure if it is working or not. Lower amount of traffic but we were getting killed in publish after replications as well when the dispatcher would flush: now.
                Translate
                Kunal_Gaba_
                Kunal_Gaba_
                New Participant
                Dec 18, 2020 3:32 PM UTC

                Can you check in the logs if there are any session leaks or if there are any slow unresponsive queries ? Do you see any errors in the logs ?  What do you see in health check dashboards ? Any patterns of high memory or CPU consumptions or disk utilizations ?

                Translate
                  sdouglasmcSonovaAuthor
                  Level 4
                  Dec 18, 2020 3:40 PM UTC

                  ...

                  Translate
                  Terms & ConditionsAccessibility statement

                  Loading footer...