Your browser does not support JavaScript! Skip to main content
Free 30-day trial Customer portal Contact
 
Rapita Systems
 

Industry leading verification tools & services

Rapita Verification Suite (RVS)

  RapiTest - Unit/system testing   RapiCover - Structural coverage analysis   RapiTime - Timing analysis (inc. WCET)   RapiTask - Scheduling visualization   RapiCoverZero - Zero footprint coverage analysis   RapiTimeZero - Zero footprint timing analysis   RapiTaskZero - Zero footprint scheduling analysis

Multicore verification

  CAST-32A Compliance   Multicore Timing Solution   RapiDaemons

Services

  V & V Services   Qualification   Training   Tool Integration  Support

Industries

  Aerospace (DO-178C)   Automotive (ISO 26262)   Space

Other

  RTBx   Mx-Suite   Software licensing   Product life cycle policy

Latest from Rapita HQ

Latest news

Rapita hosts SCADE webinar with ANSYS
FACE Virtual Technical Interchange Meeting
RVS 3.14 Launched
Propelling the next generation of scientists
View News

Latest from the Rapita blog

Software verification on the Solar Orbiter
Metrowerks CodeTest - How and why to upgrade
Leveraging FACE Conformance Artifacts to Support Airworthiness
Assured Multicore Partitioning for FACE Systems
View Blog

Latest discovery pages

matlab_simulink MATLAB Simulink MCDC coverage and WCET analysis
code_coverage_ada Code coverage for Ada, C and C++
amc-20-193 AMC 20-193
embedded_software_testing Embedded Software Testing Tools
View Discovery pages

Upcoming events

Future Airborne Capability Environment (FACE) Exihibition
2021-09-14
Safe Use of Multi-Core Processors Seminar
2021-11-10
View Events

Technical resources for industry professionals

Latest White papers

Multicore Timing Analysis for DO-178C
Seven Roadblocks to 100% Structural Coverage (and how to avoid them)
Eight top code coverage questions in embedded avionics systems
View White papers

Latest Videos

Efficient testing with RVS and ANSYS® SCADE® Test™
Enabling cost-effective modular avionics with FACE
Streamlined software verification thumbnail
Streamlined software verification with RVS 3.14
Qualification guidance thumbnail
Clear qualification guidance with RVS qualification kits
View Videos

Latest Case studies

Cobham Aerospace Connectivity: RapiCover continues to deliver on the most challenging targets
DO-178B Level A Embraer FCS
Validation of COTS Ada Compiler for Safety-Critical Applications
View Case studies

Other Downloads

 Webinars

 Brochures

 Product briefs

 Technical notes

 Research projects

Discover Rapita

Who we are

The company menu

  • About us
  • Customers
  • Distributors
  • Locations
  • Partners
  • Research projects
  • Contact us

Careers

Careers menu

  • Current opportunities & application process
  • Working at Rapita

US office

+1 248-957-9801
info@rapitasystems.com
41131 Vincenti Ct.
Novi, MI, 48375
USA

UK office

+44 (0)1904 413945
info@rapitasystems.com
Atlas House
York, YO10 3JB
UK

Back to Top

How to trace the source of deadlocks

Breadcrumb

  1. Home
  2. Blog
  3. How to trace the source of deadlocks
2014-08-28

In this article I explain how you can use the principles of tracing tools to find the sequence of events that lead to a deadlock. A deadlock is a cyclic dependency, for example where action A cannot continue because it is waiting for a resource to be unlocked by action B, however action B is waiting on A before continuing.

Deadlocks are one of the many banes of multi-threaded development for dynamically scheduled systems. Even though the locations of the deadlock itself are easy enough to find, establishing how we got to that point is often a harder task.

Access to resources is managed by access control locks, such as semaphores or mutexes. In figure 1 (below) this is represented by a chain of tasks claiming locks and locks claimed by tasks. Task and lock can only be in one of three dependency states:

  • Task is waiting to claim a lock (task depends on lock)
  • Lock is claimed by a task (lock depends on task)
  • Task is not waiting and lock is not claimed (no dependencies)
A chain of tasks claiming locks and locks claimed by tasks

Figure 1: A chain of tasks claiming locks and locks claimed by tasks

Knowing the state of each task and lock combination in the system is enough to confirm a deadlock and show the sequence of locks that lead up to this.

Note that, when dealing with binary semaphores and mutexes, these dependency relationships are a many-to-one relationship. This makes the state-machine of the relationships deterministic. So, at any point in a system, taking a snapshot of this state machine leaves us with the job of traversing this state machine to establish if there are any loops. Such a loop indicates a cyclic dependency and therefore a deadlock.

A cyclic dependency

Figure 2: A cyclic dependency

Implementing tracing functions

The first task in establishing a deadlock is to be able to record the state of the system. We can do this by wrapping any instance of lock_claim and lock_release functions with wrappers. The wrappers store key trace data at several key stages for both an attempt to lock and release. This data includes the state of the last lock attempt, the task ID and the lock ID

  • Prior to a lock attempt as WAITING
  • After the lock attempt, logging the outcome as either CLAIMED or RELEASED

One implementation option is to maintain the state machine of dependencies in memory. However, this can be costly in terms of time for something that should ideally operate at high-speed. Also, only recording the last dependency link for each object can mean that the history of execution can be lost.

In this situation, a better option could be to produce a trace. Every semaphore operation is logged so all the state machine history is preserved. A deadlock can be detected by using the trace to reconstruct the state machine of the resource locks and to detect the cyclic dependency indicating the deadlock.

Although we haven't considered the case of timeouts here, this could be handled by timestamping the trace entries. Without special handling, a deadlock that occurs due to a timed lock request (i.e. one which will time out) will still be detected. This is not necessarily a problem, even though this may be considered a false-positive, as requests that are timing out due to deadlocks in the system are not typically thought of as healthy.

There are many ways to implement the trace recording and collection, for example: logging to a memory buffer, writing to an I/O port or using a tracing debugger.

It's worth noting at this point that RapiTask, provides a framework for this type of instrumentation and data collection, which can make it easier to collect the information necessary to perform this type of analysis.

White papers

Related blog posts

Software verification on the Solar Orbiter

.
2021-03-01

Out of the box RVS integration for DDC-I's Deos RTOS

.
2020-02-23

Lightweight instrumentation with RapiTask

.
2018-06-05

What's next for Ada?

.
2014-07-04

Pagination

  • Current page 1
  • Page 2
  • Next page Next ›
  • Last page Last »
  • Solutions
    • Rapita Verification Suite
    • RapiTest
    • RapiCover
    • RapiTime
    • RapiTask

    • CAST-32A Compliance Package
    • Verification and Validation Services
    • Qualification
    • Training
    • Integration
  • Latest
  • Latest menu

    • News
    • Blog
    • Events
    • Videos
  • Downloads
  • Downloads menu

    • Brochures
    • Webinars
    • White Papers
    • Case Studies
    • Product briefs
    • Technical notes
    • Software licensing
  • Company
  • Company menu

    • About Rapita
    • Customers
    • Distributors
    • Industries
    • Locations
    • Partners
    • Research projects
    • Contact
  • Discover
    • AMC 20-193
    • What is CAST-32A?
    • Multicore Timing Analysis
    • MC/DC Coverage
    • Code coverage for Ada, C & C++
    • Embedded Software Testing Tools
    • Aerospace Software Testing
    • Automotive Software Testing
    • Certifying eVTOL
    • DO-178C Testing
    • WCET Tools
    • Worst Case Execution Time
    • Timing analysis (WCET) & Code coverage for Matlab Simulink

All materials © Rapita Systems Ltd. 2021 - All rights reserved | Privacy information Subscribe to our newsletter